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ABSTRACT 
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student inputs f course content , and training outcome variables. 
Measures of these variables were formulated on the basis of personnel 
records and routinely available training documents , end data were 
assembled for over 5,000 trainees in 39 initial-skills courses at 
five Air Force bases. A cross-validation sample consisted of about 
1,000 trainees in nine courses. Using formal path analysis (LISREL V 
computer program), an empirical model of the training process was 
developed. Although aptitude was one of the best predictors of the 
quality of student performance, it interacted with a variety of other 
student input and course content variables in determining training 
outcomes. Course content variables tended to be defined by course 
subject-matter difficulty, occupational difficulty, and manpower 
requirements, Counseling mediated the relationship between training 
performance and distal training outcomes such as retraining time and 
student elimination. Appendices present sample interview questions, 
an Air Force policy document on student measurement (with reporting 
forms), and tables of input statistics and path analysis results. 
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SUMMARY 



The objective of this research and development (RW) effort was to dovelop and validate an 
empirical procedure for predicting training outcomes associated with initial-skills courses 
administered by the United States Air Force Air Training Command. The RZO consisted of the 
collection and analysis of information to determine the effects of student attributes and course 
properties on training outcomes such as academic training performance and student elimination. 
The value of the procedure resides 1n its use as a decision support system for personnel and 
training management. It can be applied on a course-by-course basis to evaluate "what-if" 
questions concerning the effects of changes in student and course characteristics cn training 
outcomes. For example, the system could be used by personnel managers to forecast the training 
outcomes likely to result from decreases in the minimum aptitude test score required for entry 
Into an Initial-skills course. It could be used by training manager: to forecast the training 
outcomes likely to result from increases or decreases in the trained personnel requirement for 
the course. It could be used by course designers to forecast the training outcomes likely to 
result from adjustments 1n course length or student/Instructor ratio. In addition to providing a 
means of forecasting training outcomes associated with changes in single variables, it could be 
used to forecast training outcomes associated with simultaneous adjustments in several student 
and course characteristics. ' The p^off of this RID is a decision support system for personnel 
and training management that can be applied to anticipate and avoid adverse training outcomes. 
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PREFACE 



This R&D is one of several efforts conducted in support of Request for Personnel 
Research (RPR) 73-17, Minimum Aptitude Requirements for Air Force Enlisted Specialties. 
The primary objective of research conducted in support of the RPR was to develop an 
empirical procedure for evaluating aptitude requirement minimums for enlisted 
specialties. Such a procedure was developed and is described in an AFHRL technical 
report by Burtch, Lipscomb, & Wissman (1982) and a special report by Weeks (1984). The 
RID described in this report extends the focus of the research to include development of 
a system for assessing the impact of aptitude requirement adjustments on initial-skills 
training. The emerging system is potentially useful not only for avoiding the adverse 
effects on training outcomes of changes in student aptitudes but also for avoiding the 
adverse effects of changes in initial-skills courses. The research was accomplished 
under work unit 7719-19-10 and supports the Manpower, Personnel, and Training 
Integration subthrust and the Manpower and Force Management thrust. 

Credit is due the Air Force Manpower and Personnel Center, Directorate of 
Assignments, Skills Management Division, USAF Classification and Training Branch 
(AFMPC/DPMRTC) , Randolph AFB, TX, for support of the project under RPR 73-17. The 
research could not have been conducted without the generous support of the Air Training 
Command (ATC). Resources provided by ATC and authorization to conduct the R&D are 
documented in ATC Training Research Agreements #120 and #136. Because so many 
individuals participated in the project, space limitations do not permit mentioning them 
all by name. Training personnel located at each of the following organizations 
participated in the project: Air Force Military Training Center (AFMTC), Lackland AFB, 
TX; the 3700th Technical Training Wing (TCHTvJ, Sheppard AFB, TX; the USAF School of 
Health Care Sciences, Sheppard AFB, TX; the 3400th TCHTW, Lowry AFB, CO; the 3330th 
TCHTW, Chanute AFB, IL; and the 3300th TCHTW, Keesler AFB, MS. Credit is especially due 
course instructors and instructor supervisors for taking time from their busy schedules 
to provide information during initial interviews. Credit is also due personnel of the 
registrars office at each training wing for their assistance in obtaining student 
training records. Finally, credit is due Headquarters ATC, Deputy Chief of Staff 
Technical Training, Plans Directorate, Research and Technology Division (HQ ATC/TTXR) 
for assistance in coordinating the data collection effort. 

This report has been produced while R&D continues under work unit 7719-19-10 for 
the purpose of developing a computer-based system to facilitate application of the 
model. User-friendly software which implements the varioui parameters of the model has 
been developed under the acronym TTIM (Technical Training Impact Model). This effort 
will be documented in subsequent reports. In addition, final test and evaluation 
efforts are being coordinated to serve as a basis of transitioning the technology to HQ 
ATC. 
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AN EMPIRICAL SYSTEM FOR ASSESSING THE IMPACT 
OF APTITUDE REQUIREMENT ADJUSTMENTS ON AIR FORCE 
INITIAL SKILLS TRAINING 



I. INTRODUCTION 

Background 

Although many factors Influence the effectiveness of an organization, It has long been 
recognized that Individual performance Is one of the most Important determinants of goal 
attainment In any group undertaking. As a result, a wide variety of studies have been conducted 
1n an attempt to delineate the variables that Influence Individual performance, and to determine 
how these variables Interact with situational constraints to influence organizational effective- 
ness. Among the variables examined In these studies, the allocation of Individuals to jobs has 
proven to be of special significance, due to the fact that Individuals differ markedly In their 
willingness and capability to perform various tasks (Dunnette, 1966). For Instance, Schmidt, 
Hunter, MdCenzle, and Muldrow (1979) have shown that a single organization may save Itself 
millions of dollars each year by assigning to jobs only those Individuals who possess the 
attributes necessary to satisfactorily perform those jobs. 

Recognition of the fundamental Importance of optimal manpower allocation has led many 
organizations to Invest substantial time and energy In the development of decision rules for 
assigning Individuals to jobs. Typically, this Is accomplished through the use of standard 
selection Inventories to assess the extent to which an Individual possesses the attributes 
required for adequate job performance. However, In other Instances, It has been necessary to 
employ a more general and complex classification system In which the Individual 1s considered for 
assignment to a number of alternate jobs on the basis of his/her attributes and organizational 
needs (Magnuson, 1966). Application of the more complex classification paradigm for manpower 
allocation has generally been found to be most useful when the organization must obtain adequate 
performance 1n a large number of jobs within a finite timeframe and manpower pool. 

The Air Force 1s perennially faced with conditions that underscore the need for effective 
classification. Because Air Force technical training is expensive and time-consuming, „nd 
because 1t constitutes a hurdle that most airmen must pass before they can be assigned to Air 
Force specialties (AFSs), classification decisions have been based, In part, on expected training 
success. In this regard, 1t should be noted that effective allocation of individuals with 
respect to job training programs is not associated solely with short-term benefits. As Maginnis, 
Uchima, and Smith (1975a, 1975b, 1975c) have pointed out, inappropriate allocation of Air Force 
enlistees to training programs may result in a number of costly long-term outcomes ranging from 
poor job performance to high turnover. When these negative outcomes of inappropriate job 
classification decisions are aggregated over tens of thousands of Individuals, they can have a 
substantial Impact on Air Force efficiency and mission readiness. 

Personnel Allocation 1a the Air Force 

Given the foregoing concerns, It is hardly surprising that a cohesive classification strategy 
has emerged 1n the Air Force. As implied by the previous discussion, any effective personnel 
allocation system requires two central pieces of information: (a) a definition of the 
performance-relevant attributes possessed by the individual, and (b) a system for relating 
personnel attributes to effective job performance. 

In the Air Force, attributes of the individual are assessed primarily through the Armed 
Services Vocational Aptitude Battery (ASVAB) (Weeks, Mulllns, & Vitola, 1975). The ASVAB is a 
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cognitive abilities test battery composed of a set of aptitude tests rangiug from general verbal 
and quantitative measures to specialized measures of technical knowledge. The Air Force combines 
the tests contained in the ASVAB to form four aptitude test composites. These four aptitude 
composites are designed to measure an Individual's aptitude for Mechanical, Administrative, 
General, and Electronics job specialty areas. The ASVAB aptitude composites display 
reliabilities In the mid-. 80s and validities against a criterion of technical training success In 
the mid-. 60s (Weeks, Mulllns, & Vitola, 1975). The scores on the aptitude composites constitute 
the principal measures employed by the Air Force for describing characteristics of the Individual 
likely to Influence subsequent training and job performance. 

Although ASVAB aptitude composites constitute the principal description of the Individual 
used in the Air Force manpower personnel allocation system, other descriptors are employed as 
well. For instance, the educational level and the educational preparation of the enlistee are 
considered, as are expressed Interest for a given specialty and the Individual's physical 
capabilities. In addition, access to certain specialties may be limited by manpower requirements 
and administrative requirements, such as obtaining a security clearance. 

The system the Air Force employs for relating performance on the ASVAB to organizational 
needs Is relatively straightforward. Essentially, It Is based on the concept that the most 
talented Individuals should be assigned to the most demanding AFSs. In the Air Force context, 
talent for various specialties is defined In terms of the Individual's scores on the ASVAB 
aptitude composites. Thus, under Ideal conditions, Individuals with high scores on the 
Electronics composite would be assigned to the most difficult electronics specialties. One way 
the Air Force has attempted to Implement this assignment rule Is by specifying mlnlmtm aptitude 
requirements for entry Into each AFS. 

For the personnel allocation strategy described above to serve Its Intended purpose, aptitude 
requirement mlnlmums must correspond to the occupational demand or difficulty of job 
specialties. In setting aptitude requirement minimum, technical training attrition rates have 
served as an Index of difficulty. Although this Index of difficulty appears reasonable, changes 
in training programs and manpower requirements, as well as more general administrative concerns, 
have led to changes in technical training attrition rates and, therefore, aptitude requirement 
minimums that are not necessarily in line with the difficulty of the job. For example, Weeks 
(1984) found a less-than-perfect relationship between the aptitude mlnlmums assigned AFSs and 
experts' Judgments of job difficulty. Because the optimal allocation of talent depends on a 
close match between aptitude requirement mlnlmums and the relative difficulty of job specialties, 
misalignments between aptitude requirements and job difficulty lead to assignment errors, and to 
corresponding effects on Air Force mission readiness. 

Defining Occupational Difficulty 

The definition of occupational difficulty represents a salient problem. First, it Is clear 
that training attrition rates can produce only an approximate measure of difficulty since they 
reflect training requirements as opposed to job requirements. Second, attrition rates are 
Influenced by a variety of training-specific factors (such as the appropriateness of 
Instructional materials or the availability of faculty resources) that may result in a biased 
index of difficulty. Thus, it appears that in Implementing its manpower allocation system in t.*e 
most effective manner possible, the fundamental problem facing the Air Force 1s to find a more 
appropriate measure of relative occupational difficulty. 

The fundamental Import of this problem and its potential Impact on the overall effectiveness 
of the Air Force, via Its implications for personnel allocation, led to the requirement for the 
Air Force Human Resources Laboratory (AFHRL) to undertake an extensive series of research and 
development (RU)) efforts intended to establish a more general and accurate measure of the 



relative difficulty of enlisted job specialties. Although there are obviously many potential 
2£l ta. fZ. i , att f?i tin9 t0 aSSCSS thC d1ffiCUlty ° f av career field « « believe that 
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Occupational learning difficulty is defined as the time it takes to learn to perform an 
^ ti0 " satisfactoril > ( Meeks . 1984). The construct is based on the premise that, with 
ability held constant, individuals will take a longer time to learn to perform a more difficult 
occupat onal task than a relatively easy one (Christal, 1976). This appears to be a well-founded 
assumption. In fact, substantial support for this proposition may be found in the research 

iterature with respect to its corollary that more capable individuals will take less time to 
learn a task of constant difficulty (Cronbach i Snow, 1977). Moreover, the recent cognitive 
processing literature also strongly supports the proposition that time-to-mastery and response 
time may be excellent indicators of problem difficulty at both the individual and the aggregate 

evels (Sternberg, 1982). Thus, it appears that there is sufficent evidence in the literature to 
indicate that the time taken by an individual to learn a task is an appropriate definition of 
aiTTlcu ity. 

If it is granted that "time to learn" provides an adequate index of the difficulty of an 
activity then the next question to be answered is how this conceptualization of difficulty might 
be translated Into an operational measure. It, addressing this problem, the Air Force employed a 
job analysis approach In which the time It takes to learn to satisfactorily perform occupational 
tasks Is evaluated by subject-matter experts (Christal, 1976). More specifically, AFHRL 
developed a set of benchmark rating scales against which the learning difficulty of the tasks 
performed In specialties within the Mechanical, Administrative, General, and Electronics job 

SIS n «,h rCaS f°i U J d b V valuated < F »9"1. 1972 . 1973). Subsequently, experts evaluated tasks 
within each specialty against the appropriate benchmark rating scale, and a weighted composite of 

S ! ,7 t1n9S W " obtained t0 determine the overall learning difficulty of the specialty 
(Weeks, 1981). Research has shown that these task ratings are highly reliable and display 
considerable convergent validity (Burtch, Lipscomb, * Wissman, 1982). 

The foregoing Investigations indicate that it Is possible to employ ratings of the learning 
difficulty of occupational tr-.ks in evaluating overall occupational difficulty. Moreo*»r this 
research demonstrated that measures of occupational learning difficulty could be obtained through 
?J5 strai 9 htforward extension of the Air Force's current job analysis system (Christal, 

1976). Consequently, it appeared that the occupational difficulty measure might provide a fully 
adequate and administratively feasible basis for defining the relative difficulty of 
occupations. If this measure were in turn used as a frame of reference In establishing job 
aptitude requirement minimums, it was felt that It might Improve the current personnel allocation 
system. However, preliminary analyses (Weeks, 1984) have indicated that the realignment of 
aptitude minimums in accordance with occupational learning difficulty would lead to marked shifts 
In aptitude minimums for several specialties. 

The Problem 

A wide variety of studies have indicated that scores on aptitude measures are directly 
related to educational performance. For Instance, Tyler (1964) summarized a number of studies 
and concluded that general cognitive ability has a marked Impact on training performance, such 
„nat higher aptitude students not only perform better on achievement tests but also seem to 
profit more from education. Similarly, in the Air Force context, Wilbourn, Valentine, and Ree 
(1984) found that performance on the ASVAB Is substantially related to training performance in a 
number of different occupational fields. 
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' Obviously, Intellectual aptitude Is not the only variable that can influence training 
performance* For Instance, It has been pointed out that Interest, general achievement 
motivation, specific educational preparation, reading ability, maturity, study habits, and 
self-discipline, among other variables, may also affect training performance (Frederlckson & 
Gilbert, 1964; Gough & Fink, 1964). Moreover, as Terman (1959) pointed out, all these variables 
interact with general cognitive ability in subtle but generally positive ways. As a result, it 
Is critical that any effort to align aptitude requirement mlnlmums with occupational learning 
difficulty Include careful consideration of the Impact of such aptitude requirement adjustments 
on technical training outcomes. Further, one must consider not only the direct effects of 
aptitude but also the indirect effects that aptitude has on other individual attributes related 
to training performance. 

In considering the potential Impact of aptitude requirement adjustments on training 
performance, there is another set of variables that should be considered. Currently, the Air 
Force formulates Instructional programs through a variation of the Instructional systems 
development (ISO) paradigm proposed by Goldstein (1974) and elaborated by O'Nell (1979) among 
others. As with most well-designed instructional systems, the characteristics of the student 
population are carefully examined, including educational preparation, reading ability, aptitude, 
and motivation. Thus, the Indirect Impact of aptitude requirement adjustments on technical 
training might be manifested not only through other attributes of the Individual, but also 
through the design of training materials and course variables such as the length of training. 

It should also be recognized that the Impact of aptitude on training performance might itself 
be moderated by course content variables, and that aptitude may Interact with these variables In 
determining training outcomes. For Instance, Cronbach and Snow (1977) reviewed different sources 
of evidence that, while not conclusive, do suggest that features of the Instructional process may 
Interact with aptitude and other individual attributes in determining training outcomes. It 
should be noted here that similar effects have been observed in the military setting in studies 
conducted by Fox, Taylor, and Caylor (1969), as well as Federlco and Landis (1979). 

Taken as a whole, the preceding discussion indicates that changes in aptitude requirement 
mlnlmums will lead to changes In training performance. However, it Is difficult to specify the 
exact nature and magnitude of these changes since the effects of aptitude may be moderated by 
other individual attributes and by various characteristics of the particular training program. 

This ambiguity was of sufficient concern to Air Force training and personnel managers that it 
was felt that before any major adjustments were made In aptitude requirements on the basis of 
occupational learning difficulty, RM> should be conducted to provide some insight into the Impact 
of such aptitude requirement adjustments on technical training outcomes. Finally, it was argued 
that the Investigation should carefully consider those complex attributes of both students and 
training program that might Interact with aptitude in determining training outcomes. The 
present investigation was undertaken in an attempt to meet these goals. 



II. METHOD 

Overview 

A variety of different methodological strategies might be employed to address the concerns 
outlined previously. However; the nature of Air Force initial-skills training, along with the 
goals of the present effort and the pertinent research literature, Indicated the need for a 
particular methodological and conceptual approach. In reviewing the relationship between 
cognitive abilities and training outcomes, it became apparent that the effects of aptitude on 
training outcomes could adequately be captured only by considering a variety of student 
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attributes related to aptitude, such as academic motivation and educational level, as well as the 
IL inte " ction of ^itude and associated variables with aspects of the training process 
associated with a given specialty. Moreover, Kirkpatrick (1959) noted that multiple training 
outcome variables must be examined in any attempt to assess training performance. For example 
when examining technical training outcomes, it is clear that the quality of student performance 

r ol\l°l ^ 35 WCl1 " StUdCnt attn ' ti0n rates * These observations taken as a whole 
suggest that any attempt to address the issues at hand must take a multivariate approach capable 
of Incorporating a variety of student input, course content, and training outcome variables. 

In addition to a multivariate approach, practical considerations require that the 
^^ h I PS e T S h V bliShed *» n * these variables must be capable of being used in making routine 
predictions. This requires that measures of these variables be applicable to all technical 
training courses under consideration. When this observation was coupled with the fact that R«0 
examining training outcomes would be feasible only to the extent that it did not disrupt ongoing 
training efforts, a correlational field study seemed the most appropriate approach. Finally 
because the Air Force Air Training Command (ATC) currently administers approximately 200 
resident, initial-skills training courses, it was apparent that t.ie student input, course 
content, and training outcome variables, as well as their interrelationships, could not be 
established separately for each course. Thus, it seemed necessary to employ general descriptive 
variables 1n each of these areas and attempt to establish typical relationships across a variety 
of initial-skills courses. 

In view of these considerations, it was decided to use the modeling strategy of path 
analysis, which is an extension and generalization of the traditional regression model "and which, 
Hke the regression approach, allows predictions in such a way that permits multiple causal 
relationships to be considered. This decision was supported by the fact that multivariate models 
have proven useful in addressing a variety of complex problems in the social sciences, ranging 
from interest development (Ekehammar, 1977) to the determinants of collegiate academic 
performance (Ribich 4 Schneck, 1979). An excellent overview of this diverse literature, as well 
as Illustrations of the potential applications of this approach 1n addressing problems involving 
multivariate prediction, may be found in Joreskog and Sorbom (1980). 

After 1t was determined that a multivariate modeling approach would be most appropriate tor 
addressing the problems at hand, the next issue to be resolved was how this approach should be 
implemented. Generally speaking, the use of a multivariate modeling approach is contingent on 
the formation of a sound theoretical framework (Kenny, 1979). This theoretical framework in turn 
serves as a guide for determining the nature of the variables that should be included in the 
model, as well as their hypothetical interrelationships. The model is then tested by contrasting 
theoretical expectations, and the variance-covarlance matrix they imply, against an empirical 
varlance-covanance matrix derived from a set of observations. If the theoretical model fits the 
data, one can have confidence 1n its appropriateness. The weights generated in fitting the 
theoretical model to the observed data may then be used as a basis for prediction. 

The foregoing overview of the path analysis served to outline the general methodological 
steps employed 1n the present Investigation. First, a general conceptual framework for 
understanding the relationships among student inputs, course content, and training outcomes was 
formulated. Second, this theoretical framework guided the specification of the variables likely 
to Influence training performance. Third, adequate measures of these variables were developed or 
obtained. Fourth, the measures were applied to a number of different individuals in a 
representative sample of Initial-skills courses. Fifth, the observed relationships among these 
variables were evaluated 1n light of the model. Of course, modifications were made to this 
general methodological framework as a result of practical constraints. For instance, 
investigators commonly modify their Initial theory once correlational data become available if 
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any limitations In their original theory become apparent. Nevertheless, these steps constitute 
the general method employed In the present investigation. 



The Conceptual Model 

The preceding section suggested some ways aptitude can influence performance in training. 
This section will develop this relationship In more detail, starting with the most 
straightforward relationship, which assumes that aptitude directly causes training performance. 
Model A of Figure 1 schematically depicts this relationship. Although this may be an appropriate 
conceptualization in situations where the aptitude measures are heavily weighted toward prior 
achievement, in the context of the present effort, It is unlikely that such a simple, direct 
relationship exists, since the ASVAB is a general measure of intellectual ability and most Air 
Force enlistees have little prior experience with the work required in the specialties for which 
they are trained. 

Model A 
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Figure 1 . Initial Conceptual Models. 

These observations suggested that some fundamental intervening processes must link aptitude 
to training performance. There can be little doubt that this Intervening process is learning. 
This extension of the simple, direct linkage model is presented in Figure 1 as Model B. In this 
case, aptitude facilitates ease of learning, which In turn facilitates training performance, 
given a constant training time. This extension of the initial model Is supported in a literature 
review by Cronbach and Snow (1977), who summarized a number of studies Indicating that aptitude 
generally leads to more rapid attainment of training performance standards. Moreover, additional 
support for the extended model is suggested by Its alignment with the theory underlying the 
occupational learning difficulty research conducted by AFHRL. 

When ease of learning Is conceived of as a hypothetical variable intervening between aptitude 
and performance, the significance of, and need for, a second class of variables becomes 
apparent. One of the principles underlying Instructional systems development is that performance 
in training can be enhanced by the use of effective training strategies (Goldstein, 1974). This 
suggests that variables that characterize the training program and that might influence ease of 
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learning, such as quality of Instruction and amount of feedback (Latham & Wexly, 1980), should be 
Included 1n the model. Model C presents a revision of Model B which incorporates training 
variables. However, it should be recognized that, as well as having an impact on ease of 
learning, these course content variables may also have a separate impact on performance. For 
example, Instructor-student interactions may affect evaluations of student performance, in 
addition to having an impact on ease of learning. Thus, it was necessary to consider the 
possibility that course content variables may have a direct impact on performance. Moreover, it 
should be noted that because initial-skills training courses are designed in relation to the 
student population, aptitude itself may also have a direct impact on these course content 
variables. These considerations led to Model 0. 

Support for Model 0 1s also found in the literature. This model postulates a complex set of 
interactions between aptitude and course content variables and holds that both classes of 
variables may have direct effects on performance. Phenomena of this sort have been observed in a 
variety of studies Including those conducted by Smith (1942) and Newman, Freeman, and Holzlnger 
(1937). Yet, despite the existence of such support, this version of the conceptual model is 
still Incomplete. The literature reviewed earlier suggests that any effects of aptitude on 
training performance may be expressed through or moderated by other student inputs such as 
reading ability or academic motivation, which are themselves related to aptitude. As a result, 
it seems that the aptitude component of the model should be expanded to include a variety of 
performance-relevant attributes of the individual. However, it should be recognized that 
variables such as motivation may themselves have a direct impact on performance which is 
Independent of their impact on ease of learning. Finally, it should be noted that although 
training performance represents an abstract construct, it is manifested in a variety of training 
outcomes which, in some way, have cost implications for the A1r Force. 

The foregoing considerations led to a further revision of the initial conceptual model. This 
model, presented in Figure 2, represents the general conceptual model employed in the present 
investigation. As may be seen, both course content variables and student inputs were held to 
have positive causal effects on training outcomes, as Indicated by the plus signs and arrows. 
Student input and course content variables were also held to be capable of having a direct causal 
effect on training outcomes regardless of ease of learning. Finally, ease of learning was 
assumed to have a direct positive causal effect on training outcomes, and adverse training 
outcomes were assumed to have a negative effect on training costs. Basically, the general 
conceptual model indicates that training outcomes are a function of student inputs and course 
content, and that these outcomes 1n turn affect training costs. This rather straightforward 
conceptualization provided the preliminary theoretical framework required to develop an empirical 
model of the relationship between aptitude and training outcomes. 
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Figure 2 . Refined Conceptual Model- 
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Variable Specification 



Following development of the general conceptual model, the next step 1n this effort Involved 
identifying the student Input and course content variables capable of Influencing ease of 
learning, as well as the likely relationships among these variables. Once these variables had 
been Identified, 1t was then necessary to generate measures of these variables and ensure the 
relevance of these measures to training performance. In order to obtain this Information, it was 
necessary to conduct a series of interviews with A1r Force personnel directly Involved in the 
technical training process. 

A set of preliminary Interviews was conducted at Headquarters Air Training Comnand (HQ ATC) 
by two Advanced Research Resource Organization (ARRO) staff members and two representatives of 
AFHRL. Interviewees were drawn from organizational groups charged with the overall control and 
coordination of technical training. These Interviews were not highly structured but served to 
provide an overview of the technical training process. Additionally, at this time, a sample of 
documentary materials were obtained including: 

1. ATC Forms 156, Student Record of Training, 

2. plans of instruction (POIs),- 

3. specialty training standards (STSs), and 

4. course charts (CCs). 

The information obtained in the preliminary interview was then used to design a more 
'structured interview protocol to elicit information pertaining to: 

1. variables Indicative of the quality of student training performance; 

2. student input variables that might influence training performance quality; 

3. course content variables that might influence training performance quality; 
4* probable relationships between course content and student input variables; 

5. potential measures of student Input, course content, and training outcome variables; 

6. potential biases in these measures; and 

7. potential sources of these measures. 

An example of the interview protocol appears in Appendix A. Once this interview protocol had 
been developed, a series of visits were then made to four ATC technical training centers: Lowry 
AFB, Keesler AFB, Sheppard AFB, and Chanute AFB. At each site, interviews were conducted with: 

1. course instructors, 

2. instructor supervisors, 

3. training managers, 

4. instructional systems design chiefs, 
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5. wing educational advisor, 



6. measurement section personnel, and 

7. registrar. 

These Individuals were chosen so that several different perspectives of the technical 
training program would be represented. During the interviews, an attempt was made 1n all cases 
to focus discussion on those questions contained in the general interview protocol most relevant 
to the Interviewees 1 areas of expertise. 

The Interviews were conducted by two ARRO staff members and two AFHRL representatives during 
the winter of 1983. One ARRO staff member was responsible for guiding the discussion, while the 
other was responsible for generating a written summary of the discussion. The written sunwaries 
obtained from these Interviews were subsequently reviewed by ARRO staff members 1n an attempt to 
identify: (a) the major measurable outcomes of resident technical training, (b) the student 
inputs likely to Influence outcomes within each course (and across courses), and (c) the course 
content variables likely to Influence outcomes within each course (and across courses). 

The review also focused on (a) the Interrelationships among student Input and course content 
variables across courses, (b) the available measures of these constructs, and (c) any known 
biases 1n these measures. Once this information had been obtained, it was used to specify the 
variables that would be considered for Inclusion in the model. 



Variables and Measures 

The variables considered for Inclusion in the model were selected on the basis of information 
collected during the Interviews. Because specification of variables constitutes a critical step 
1n the development of a valid multivariate model, substantial attention is given to this topic 
below. This discussion begins with an examination of training outcome variables and then turns 
to the student inpr> and course content variables. 

Training Outcomes . To understand the significant outcomes of initial-skills training in ATC, 
1t is necessary to have some understanding of the goals of the training effort. In the Air 
Force, initial-skills training is not intended to produce a fully qualified journeyman capable of 
performing all tasks within a specialty. Rather, students are trained to a level of partial 
proficiency in the tasks they are most likely to perform upon job entry. In the Air Force, this 
is called "training to the 3-sk1ll level." The assessment of trainee performance focuses on the 
acquisition of knowledge and performance capabilities associated with this skill level. 
Assessment procedures are prescribed in ATC Regulation 52-3, S tudent Measurement Guidelines , 
This document, which is presented 1n Appendix B, serves to specify exactly how and when students 
should be evaluated. Although this regulation allows for some local variation 1n student 
evaluation procedures, Its content has led to the emergence of a relatively standard set of 
procedures. 

Typically, a technical training course is divided into units that present a cohesive body of 
material and blocks of Instruction that are composed of varying numbers of units. Within each 
unit, quizzes or progress checks (PC) that have been formally specified in the course Plan of 
Instruction (POI) are administered. Students must pass each PC before being allowed to advance 
to the next unit. If the student fails a PC, the instructor is expected to provide Special 
Individual Assistance (SIA) until the PC has been passed, although SIA may be provided for other 
reasons as well. Once the student has passed all PCs in a block, an end-of-block written test is 
administered. This end-of-block test 1s a standard paper-and-pencil academic achievement test. 
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The student must obtain a minimum score of 70 on the block test to advance to the next block of 
Instruction. If a falling grade Is obtained, the student may be required to repeat the block of 
Instruction before retaking the test. Failure to pass on the second attempt usually leads to 
student elimination. 

All student evaluations are recorded. Progress check performance Is recorded on ATC Form 98, 
Student Progress Checklist, which is destroyed once the student has completed the course. 
End-of-block tests are recorded on ATC Form 156, Student Record of Training, which Is maintained 
by the training center registrar's office as a permanent record of an Individual's training 
performance. An example of ATC Form 156 is provided In Appendix C. A student's final grade for 
the course Is the average of all end-of-block tests. The consensus of the Interviewees was that 
the average final course grade Is an adequate, if not excellent, index of the quality of student 
performance. Therefore, final course grade, as defined by average performance on the block 
tests, was used to provide an Index of the quality of student training performance In the model. 

ATC Form 156 also provides a variety of additional Information bearing on training 

performance. One such item Is hours of SIA time provided the student. Interviewees Indicated 

that SIA time was accurately recorded, and that It was a costly and Important outcome. Thus, U 
seemed prudent to consider SIA time as an additional training outcome. 

ATC Form 156 also contains information concerning two other outcomes that Interviewees 
considered to be of substantial importance. All negative actions concerning a student, such as 
elimination from training or scheduling for retraining, must be preceded by a counseling 
session. In these counseling sessions, which usually last about an hour, student performance is 
reviewed by an instructor supervisor who talks to the student and determines what action should 
be taken. The numbers of academic and nonacademic counseling sessions are recorded on the ATC 
Form 156, and it is noted whether each session was held for academic or nonacademic reasons. 
Given their importance, it seemed necessary to quantify counseling sessions as a training 
outcome. This was done by determining separately the total number of academic and nonacademic 
counseling sessions a student received during the course. 

Another potential outcome of training (i.e., an alternative index of student performance) Is 
retraining time. Retraining occurs when a student Is required to repeat one or more blocks of a 
course as a consequence of falling an end-of-block test. Tills outcome Is significant In part 
because retraining Is remedial action taken prior to elimination for academic reasons, and In 
part because retraining is quite expensive. A measure of retraining time m^ be obtained from 
the ATC Form 156 by subtracting the formal course length from the individual's total hours In 
training. 

As implied by the foregoing discussion, nearly all groups Interviewed Indicated that two of 
the most Important training outcomes are academic and nonacademic elimination. Both academic and 
nonacademic eliminations are directly recorded on ATC Forms 156. This information was used to 
code as 0's students who graduated and to code as l's students who were eliminated from training 
for either strictly academic or strictly nonacademic reasons (not administrative reasons such as 
death or lack of a security clearance). Students who were eliminated presented a special problem 
with respect to the measurement of SIA time, retraining time, academic counseling, nonacademic 
counseling, and the quality of student performance. The problem was to estimate expected values 
for these variables which probably wpuld have been observed If the student had remained In the 
course for the entire training period. The expected value of quality of student performance was 
estimated by taking the average of all block test grades up to the point of elimination. For SIA 
time, retraining time, academic counseling, and nonacademic counseling, the expected values were 
estimated by calculating the rate per training hour up to the point of elimination and Increasing 
the observed value at the point of elimination by the product of training hours not completed and 
the rate per training hour. 
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Overall, seven training outcomes were identified as being significant variables: assessed 
quality of student performance, SIA time, retraining time, academic counseling, nonacademic 
counseling, academic attrition (i.e., student elimination), and nonacademic attrition. Measures 
of all these variables could be obtained from the permanently maintained ATC Form 156. Although 
performance on PCs would seem to be a potentially significant outcome variable, this information 
was no longer available* Based upon these observations, coupled with interviewees 1 comments that 
the student evaluation data were accurately recorded and represented a comprehensive description 
of training performance, It seemed reasonable to conclude that these measures would provide an 
adequate basis for model development. 

Student Inputs . Practically all Interview groups agreed that a student's aptitude 
significantly Influences training outcomes. In the Air Force, at least three measures of 
aptitude are available: Armed Forces Qualification Test (AFQT) score, ASVAB subtest scores, and 
scores on the ASVAB aptitude composites. Given current Air Force classification practices, it 
was decided that the ASVAB aptitude composite score should be used as the principal Index of 
aptitude. This decision also finds some support In the fact that the aptitude composite scores 
have been shown to be reliable and valid predictors of technical training performance (Wllbourn 
et al., 1984). In the present effort, although It was possible to use all four aptitude 
composite scores as Indicators of student aptitudes, it was deemed more appropriate to use only 
the aptitude: composite score that applied to the specialty for which the individual was being 
trained. 

Although all interviewees considered aptitude to be of great Importance, they also noted that 
a number of other student characteristics Interacted with aptitude In determining training 
performance. For Instance, trainers noted that reading ability often had a substantial Impact on 
training performance, due to the reading difficulty of the technical material used in certain 
courses. ^Currently, two measures of reading ability are available on the ATC Form 1J6 which 
reflect vocabulary and reading comprehension levels. These measures are derived from the Air 
Force Reading Abilities Test (AFRAT), which Is currently administered during basic military 
training. Alternate forms of this test have reliabilities ranging between the mid-. 80s and 
low-. 90s (Mathews 6 Roach, 1982). Given the availability of these measures and the perceived 
effect of reading ability on training outcomes, AFRAT scores were Included in the model as an 
average of the vocabulary and reading comprehension scores. 

In addition to aptitude and reading ability, course supervisors ana instructors emphasized 
the Importance of educational preparation because of its relation to study habits and ability to 
cope with an academic environment. Currently, the Air Force does not use direct treasures of 
educational preparation In the classification process; however, two indirect measures were 
available. The first of these was the educational level attained by the Individual prior to 
entering the Air Force. These data are recorded In the Processing and Classification of 
Enlistees (PACE) personnel data file. For the present effort, this variable was scored on a 
5-point continuum such that non-high school graduates received a score of 1, high school 
graduates received a score of 2, those with some college received a score of 3, college graduates 
received a score of 4, and those having post-graduate work received a score of 5. A second 
measure of educational preparation was also obtained from the PACE file. The PACE file contains 
a listing of some 42 different high school courses that could have been, taken by the Individual 
prior to entering the Air Force. For most specialties, the Air Force lists one to five of these 
courses as desirable prerequisites for entry Into an AFS. Thus, an additional measure of 
educational preparation could be obtained by summing the number of suggested high school course 
prerequisites that the Individual had completed prior to entering the Air Force. Although the 
reliability and validity of these particular measures were not established, available evidence 
concerning such background data measures suggests reliabilities In the mid-. 90s as well as 
substantial predictive power. 
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The high school course records contained In the PACE file also made It possible to assess 
another student Input deemed to be of some Importance to training performance. The Interview 
groups consistently underscored the Importance of student motivation in determining training 
performance. Although neither the scope of the present effort nor the current Air Force 
classification procedures would permit the assessment of general achievement motivation, It was 
possible to build a measure of academic achievement motivation from the available background 
data. This was done by having five psychologists rate the overall difficulty of each of the 42 
high school course prerequisites on a 5-potnt scale. The Interrater reliability of these 
judgments was In the low-. 90s. These ratings were used to develop an Index of academic 
achievement motivation by forming a weighted sum of the difficulty of the high school courses 
taken by an Individual. Although the reliability of this Index 1s not known, given the 
reliability of academic history data and the reliability of the difficulty ratings, It appears 
that the reliability of the Index would be fairly high. A list of the difficulty ratings 
assigned to the 42 high school courses is provided in Appendix 0. 

During the Interviews, Instructors and instructor supervisors often noted that the maturity 
of the student was jf some Importance in determining training outcomes. At least two indicators 
of maturity were available for use in the present effort. The first of these was age at the time 
of entry Into training. The second was whether the student was a new enlistee, as opposed to 
being an experienced airman retraining for a new specialty. Although the latter measure appears 
to be a somewhat more direct index of vocational maturity, the Infrequency with which experienced 
airmen are retrained into totally new specialties led to the use of age as a general Index of 
maturity. Given the high reliability of this Index, the limited age range involved, and the 
control for education level and aptitude already Included among student Inputs, inclusion of this 
measure seemed well justified. 

Students 1 interest In the specialty for which they are being trained was also cited by many 
Interview groups as having a significant influence on training performance; however, few indices 
of vocational Interest are available in the Air Force. The most appropriate of these indices is 
the Air Force Vocational Interest-Career Examination (Alley, Wilbourn, & Berberich, 1976). 
Unfortunately, this measure is not yet In operational use and so could not be employed In the 
present effort. Two alternate Indices of interest could also be Identified, however. The first 
of these was a measure of whether the trainee had or had not been guaranteed training in a 
particular specialty as part of his/her enlistment contract. Trainees having guaranteed training 
were coded as 2; all others were coded as 1. The second of these, also obtained from the PACE 
file, was a combination of information concerning the specialty to which the trainee was assigned 
and a rank-ordering of the trainee's five specialty preferences. From this information, an 
interest Index was derived by coding as a 5 trainees who got their first choice; by coding as a 3 
trainees who got their second, third, fourth, or fifth choice; and by coding as a 1 trainees 
assigned to a specialty that was not ranked. Unfortunately, when selecting specialties for 
training, enlistees have little knowledge concerning the content and nature of the specialties. 
Thus, these two vocational interest measures were considered to be of limited value, and were 
only tentatively considered for inclusion in the model. 

At least three other classes of student Input variables were mentioned as being related to 
training performance. The first class of variables Included factors such as resistance to 
stress, and attention to detail. Unfortunately, measures of these variables &re not routinely 
collected. Moreover, their impact on performance appears to be AFS-specif 1c, in the sense that 
resistance to stress was seen as being particularly Important for Air Traffic Controllers but not 
tor other specialties such as Personnel Specialists. This specificity argues against the use of 
such variables in formulating a general model of technical training; and so, for both pragmatic 
and theoretical reasons, variables of this sort were not included. A second class of student 
Inputs mentioned as being related to training performance Involved physical ability. However, 
the specificity of occupational physical demands and their limited relevance to classroom 
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PGricr.r«dnce led to the rejection this class of variables despite their use in the selection and 
classification process. Finally* demographic variables such as sex» ethnicity, and socioeconomic 
status were not considered for inclusion In the model because they have only an Indirect 
relationship to training performance. 

Despite the exclusion of these latter variables, it appears that the student input variables 
specified for Inclusion 1n the modal provide a reasonably comprehensive description of the 
relevant student attributes. Measures of the variables selected in the present Investigation 
were obtained from the PACE file, with the exception of AFRAT scores, which were obtained from 
ATC Form 156. The ready availability of these measures and their general significance to 
training appeared to provide a sound practical basis for model development. 

Course Content . In specifying the course content coi»yonent of the model, a variety of course 
variables were mentioned by Interviewees as being of some import. One of these variables was 
course length. It was often reported that time constraints were an Important consideration In 
determining training outcomes and that given sufficient time, anyone could be trained. Such 
comments Indicated that course length should be Included In the model. This variable could be 
measured by simply obtaining the number of hours of classroom instruction specified In the course 
POI. !n discussions with Instructors and course supervisors, It was also noted that course 
training. programs provided either a full 8 hours of lecture each day or 6 hours of lecture plus 2 
hours of supervised study. Generally, they felt that what was called the "Improved" or 
6-plus-2-hour Instructional day contributed to better performance because the Information load on 
students was less, thus preventing student "burnout." Thus, the length of the instructional day 
was measured by coding an 8-hour day as a 1 and a 6-plus-2-hour day as a 0. In the case of both 
Instructional day length and course length, these measures can be assumed to be perfectly 
reliable due to the highly structured and tightly scheduled nature of resident technical training. 

Another topic that received a great deal of attention In the Interviews concerned the quality 
and experience of Instructors. Course supervisors and branch chiefs pointed out that Instructor 
experience in the classroom often seemed related to the quality of student performance, as did 
the overall quality of the Instruction provided to students. 

Instructor quality Is assessed through supervisory ratings and is recorded on ATC Form 281, 
Instructor Evaluation Checklist. This form Is a behavioral checklist containing 19 
nonadmlnlstratlve Items which had an Interrater reliability (R k,c ) of .83 for a sample of 100 
Instructors drawn from 48 different courses. These Instructor evaluations are conducted at 
specified Intervals, depending on the particular Instructors experience. Performance on each 
item Is graded as good, satisfactory, or poor. These evaluations are retained as permanent 
records and could be easily obtained. Although some concern with their accuracy was voiced 
during the Interviews, It was felt that they provided a useful Index of the quality of the 
Instruction being provided In a course. Ratings on each Item were quantified by assigning good 
ratings a score of 3, satisfactory ratings a score of 2, and poor ratings a score of 1. The 
quality of individual Instructors was calculated as an average of the Item ratings, while quality 
of Instruction at the course level was determined by averaging the ratings received by all 
Instructors assigned to the course. Instructor experience presented a somewhat less complex 
variable In terms of measurement procedures since interviewees indicated that the number of 
months of Instructional experience possessed by each Instructor could be readily provided by 
course supervisors. Thus, a reliable Index of course-level instructor experience could be 
obtained simply by calculating the average months of experience of all instructors assigned to 
the course. 

For any given course, the ratio of the number of students to the number of faculty members 
was also considered as a variable contributing to student performance because the fewer students 
per Instructor, the more Instructional assistance time was potentially available to any one 



13 22 



student. It was noted that student-faculty ratio ,s a fixed course variable specified In the 
training plan (TP) developed for each course. Student-faculty ratio listed In the training plan 
was employed as a fully reliable measure of this course content variable. 

The amount of feedback and practice given to students has long been considered to have an 
effect on training performance. This point was underscored by the training personnel 
Interviewed. They pointed out that student assessment by means of PCs, and the resulting rapid 
feedback, Improved later performance jn the end-of-block written tests. The Interviewees also 
Indicated that formal feedback in terms of the progress checks and block tests was specified 1n 
the POI, and that this student evaluation schedule was rigidly adhered to,* Thus, It was possible 
to generate a reliable measure of the amount of formal feedback In a course by sunning the number 
of progress checks and block tests specified In the POI. However, It seemed necessary to control 
for course length In the development of this measure. Hence, this sum was divided by the total 
number of Instructional hours to obtain an Index of the mount of feedback per unit time. 

To address the Issue of practice, It was necessary to establish the average amount of 
training time students were given on a unified body of material. This was accomplished by 
determining the total number of units In a course and then dividing the total number of 
Instructional hours by the number of units to determine the average amount of time on each unit 
of Instruction. Again, this Information was obtained from the POI. The documentary nature of 
these data would Indicate perfect reliability for this measure, at least In the limited sense 
defined here. It should also be noted that since student evaluation in technical training 1s 
closely linked to the completion of Instructional units, it was assumed that there would be a 
strong negative relationship between practice and feedback. 

In the course of the Interviews, Instructors and Instructor supervisors often stated that 
performance could be Improved by the use of Instructional aids and hands-on practice. It was 
noted that the number of aids, as defined by job task simulations, mockups, and direct use of job 
materials, was formally specified In the course POI by unit of Instruction. Thus, a direct and 
reliable Indicator of the extent to which Instructional aids were used In a course could be 
obtained by counting the number of times mockups, job task simulations, or direct manipulations 
of job materials were required. Similarly, the POI specified the number of hands-on hours of 
Instruction by course unit and so a reliable measure of this variable could be generated by 
summing the number of hands-on hours across Instructional units. However, both the number of 
aids and the amount of hands-on training are obviously dependent on course length. To control 
for this biasing Influence, both the total number of aids and the number of hands-on hours were 
divided by course length to yield Indices of aids and hands-on practice per unit time. 

When Instructional systems design personnel, training managers, and educational advisors were 
Interviewed, It was often pointed out that AFS manning requirements constituted a significant 
Influence on the design of the associated course via a variety of direct and Indirect routes, 
ranging from their Influence on the aptitude of trainees to the frequency of feedback. Two 
Indices of marnlng requirements were obtained. First, an Index could be found In the number of 
students trained In a calendar year. Another Index of manning requirements was found In the 
availability of a monetary reenlistment bonus for the associated job specialty. Such bonuses are 
provided when there Is a need for manpower In a particular job specialty. Specialties provided 
reenlistment bonuses were coded as 1 and specialties not provided bonuses were coded as 0 in 
measuring this variable. The number of students trained per year was obtained directly from the 
course training plan. 

In addition to manning needs, the difficulty of tasks performed In a job specialty was 
frequently mentioned by instructional systems design personnel and training managers as having a 
significant Influence on the design of courses and on student performance. As a result, It 
seemed prudent to Include an Index of the difficulty of job tasks. Given the objectives of the 
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present effort, It seemed appropriate to use the occupational learning difficulty Indices 
resulting from AFHRL occupational research as a measure of the variable. Information pertaining 
to the validity of this construct Is provided elsewhere (Burtch et al. t 1982; Weeks, Mumford, & 
Hardlnq, 1985). 

As well as the difficulty of the tasks being trained, nearly all Interview groups mentioned a 
variety of other factors that Influenced the difficulty level of technical training courses. One 
variable frequently mentioned was the degree of abstractness of the course training material. 
Unfortunately, no measure of this construct was readily available; however, this concept appeared 
to be of sufflcent Importance to warrant Its Inclusion In tiie models Consequently, It was 
decided to obtain ratings of the degree of abstractness of course material. This was done by 
having five psychologists familiarize themselves with the plan of Instruction, training plan, and 
a sample of course material for each course. They then used a 5-polnt rating scale to judge each 
course In terms of the degree to which the mastery of abstract principles and concepts was 
emphasized. An average of these ratings was then used to represent the abstractness of each 
course as a whole. For a sample of 48 courses, an Interrater reliability (Rj< k ) of .88 was 
obtained for this Index. In addition, the abstractness values assigned various courses tended to 
confora to generally accepted notions concerning course difficulty. For example, the precision 
measuring equipment course received a high rating, whereas the security and law enforcement 
courses were rated lower. 

Another variable that was viewed by Instructional systems design personnel and Instructors as 
being critical to training outcomes was the reading difficulty of course materials. This was 
especially true In the more demanding training programs. Consequently, an attempt was mede to 
assess the reading difficulty of course materials. For each course, one paragraph from each of 
five different course readings was randomly selected. Subsequently, the reading difficulty of 
each paragraph was assessed using an algorithm developed by Taylor, Stlcht, Fox, and Ford 
(1973). This algorithm has been shown to exhibit reliabilities In the .90s. To compute the 
overall Index of reading difficulty for each course, the average reading difficulty level of all 
five of the selected paragraphs was derived. 

Another potential source of course difficulty Identified In the Interviews pertained to the 
diversity of course materials. It was pointed out that courses were more difficult when students 
were required to master a variety of material. This variable proved to ba measurable on the 
basis of available course Information. Because the units of Instruction within each course 
reflect a unified body of material, a fully reliable Index of diversity was derived simply by 
counting the number of units specified In the course POI. 

One final course content variable that emerged In discussions with training personnel was the 
expected student attrition rate for the course. The expected attrition rate not only Influences 
the number of students assigned to a course, but It was believed that It also Influences a 
variety of decisions In course design, as well as training outcomes such as Individual student 
attrition decisions. Thus, there seemed to be a need to obtain an adequate measure of this 
variable. This presented little difficulty because expected attrition rates art specified In the 
course training plan. Consequently, this Information could be employed to provide a highly 
reliable measure of this construct. 

For the most part, the course content variables outlined previously were those that the 
Interview groups considered to hava the greatest Influence on student performance In 
Initial-skills training. However, at least four course content variables mentioned In the 
Interviews were not Included In the modeling effort; these were student-equipment ratio, 
patterning of the academic day between lecture and discussion, swing shift versus day shift, and 
volunteer versus nonvolunteer Instructors. The latter two variables were rejected because they 
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could not be quantified without extraordinary efforts. The other variables did not appear to 
have sufficient Impact, on the basis of the Interview data, to warrant their Inclusion In the 
model. 

The preceding discussion has outlined the course content, student Input, and training outcome 
variables that were considered for Inclusion in the modeling effort, along with potential 
measures of these variables. An overview of all variables and associated measures is presented 
In Table 1. The comprehensiveness of the Interviews suggests that, for the most part, all major 
variables were included. 



Sample Selection and Data Collection 

Construction of a general model applicable to a variety of courses requires that a 
representative sample of technical training courses be examined for the purpose of model 
development and testing. Accordingly, it was decided that data would be obtained from a sample 
of 50 initial skills courses. Forty of these courses were to be used In model development while 
the remaining 10 courris would be used for cross-validation. The courses in model development 
and cross-validation were selected by AFHRL In consultation with representatives of ATC and the 
Air Force Military Personnel Center. Selection of courses was based on several criteria such as: 

1. The selected courses should be representative of all ATC Initial-skills courses. 

2. The selector aptitude composites for the courses should represent all four aptitude 
areas, but courses having dual selector composites should be excluded. 

3. There should be adequate variance In the minimum aptitude requirements for the courses 
selected. 

4. There should be adequate variance In the content of the courses. 

5. Different ATC technical training centers should be represented. 

6. Relatively high-cost courses should be included, as well as courses with high student 

flow. 

7. Courses having classified training documents (e.g., POIs) should be excluded. 
B. Recently modified courses should be excluded. 

9. Courses having computer-assisted instruction (CAI) should be represented. 

10. Courses associated with sortie-generating specialties should be represented. 

Once the sample of courses had been identified, personnel and training data required for 
model development and cross-validation were collected. The student sampling strategy was 
relatively straightforward. For each course, existing data bases were employed to identify a 
minimum of 50 individuals who had been admitted to one of the five most recent training classes. 
Subsequently, all other trainees who had entered the course during the previous 6 months were 
identified. AFHRL supplied this information along with all relevant personnel information drawn 
from the PACt file. Visits were then made to Sheppard AFB, Chanute AFB, Lowry AFB, Keesler AFB, 
and Lackland AFB, where ARRO staff members reproduced the training records of all students 
selected for inclusion in the study. At the same time, for each course, the following training 
documents were obtained: 
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Table 1 . Specification of Measures 



Training Outcome Var« J>1es 

Quality of Student 

Performance 
SIA Time 

Acadtailc Counseling 
Nonacademlc Counseling 
Retraining Time 
Academic Attrition 
Nonacademlc Attrition 

Student Input Variables 

Aptitude 

Reading Level 

Academic Motivation 
Simple Interest 
Preference Interest 
Educational Level 
Educational Preparation 

Age 

Course Content Variables 



Measure 



Average of end-of-block tests 

Hours of special Individual assistance (SIA) 
Number of academic counseling sessions 
Number of nonacademlc counseling sessions 
Number of hours of retraining 
Percentage of student academic eliminations 
Percentage of student nonacademlc eliminations 



Scores on selector *apticude composite cf 

Armed Services Vocational Aptitude Battery 
Average score on the Air Force Reading 

Abilities Test 
Number of difficult high school courses taken 
Received guaranteed specialty 
Received preferred specialty 
Highest educational level 
Recommended high school course prerequisites 

taken 
Years from birth 



Source 



Student Record 


of 


Training, 


ATC 


156 


Student Record 


of 


Training, 


ATC 


1S6 


Student Record 


of 


Training, 


ATC 


156 


Student Record 


of 


Trailing, 


ATC 


156 


Student Record 


of 


Training, 


ATC 


1S6 


Student Record 


of 


Tralnlr.g, 


ATC 


156 


Student Reccrd 


of 


Training, 


ATC 


156 



Personnel Data Files 

Student Record of Training, ATC 156 

Personnel Data Files 

Personnel Data Files 

Personnel Data Files 

Personnel Data Files 

Personnel Data Files 

Personnel Data Files 



Course Length 
Day Length 

Student-Faculty Ratio 
Instructor Expedience 
Instructor Quality 
Aids In Use 

Hands-On Instruction 

Amount of Feedback 

Amount of Practice 
Reenlistment Bonus 

Yearly Student Flow 
Occupational Difficulty 

Reading Difficulty 

Abstract Knowledge 

Expected Attrition 
Diversity 



Number of total Instructional hoixs 
Length of instructional d^y (1 equals 8* 

hour day; 0 otherwise) 
Number of students per Instructor 
Average months of Instructor experience 
Average Instructor performance evaluations 
Number of Instructional aids divided by 

course length 
Hours of hands-on Instruction divided by 

course length 
Number of evaluations divided by course 

length 

Course length divided by number of units 
1 If selective reenlistment bonus offered; 

otherwise 0 
Number of students to be trained yearly 
Overall learning difficulty of 

occupational tasks 
Average reading grade level of course 

materials 

Average rating of abstract knowledge 

requirements 
Expected proportion of students eliminated 
Number of units In course 



Plan of Instruction (POI) 
Training Plan 

Training Plan 

Plan of Instruction (POI) 

Course Supervisors 

Plan of Instruction (POI) 

Plan of Instruction (POI) 

Plan of Instruction (POI) 

Plan of Instruction (PCI) 
AFHRL 

Training Plan 
AFHRL 

ARRO Staff 

ARRO Staff 

Training Plan 

Plan of Instruction (POI) 
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1. plan of Instruction (POI), 



2. course chart (CC), 

3* training plan (TP), 

4. specialty training standard (STS), 

5. a sample of training materials, 

6. number of months of Instructional experience of instructors assigned to the course, and 

7. the two most recent instructor evaluation checklists (ATC Form 281) for all Instructors 
assigned to the course. 

An overview of the data collected may be found in Table 2. As may be seen, a complete set of 
usable data was obtained for 48 of the 50 courses identified. The courses for which complete 
data could not be obtained were associated with AFS 42334, Aircraft Pneudraullc Systems Mechanic, 
and AFS 32637, Integrated Avionics Attack Control Systems Specialist. This resulted In the loss 
of one course from both the cross-validation and model-development samples. As a result, all 
personnel and training data collected for these courses were eliminated from the analyses. 
Nevertheless, as is shown in Table 2, the remaining model -development and cross-validation 
courses were quite diverse, with titles ranging from Carpentry Specialist and Financial Service 
Specialist to Precision Measuring Equipment Specialist and Jet Engine Mechanic. In all, eight 
courses were drawn from the technical training center (TTC) at Chanute AFB t 14 from the TTC at 
Sheppard AFB, 10 from the TTC at Lowry AFB, 14 from the TCC at Keesler AFB, and two from the 
Military Training Center at Lackland AFB. Because five of the six major ATC training centers 
were represented in the present effort, representation was considered adequate. Moreover, the 
selector aptitude composites associated with the selected courses represent all aptitude areas 
and the full range of minimum aptitude requirements. In sum, these observations suggest that the 
courses identified for model development and cross-validation provide a representative sample of 
all Air Force initial-skills courses. 

For all courses combined, the student sample provided a total of 5,981 subjects for model 
development and cross-validation. All students in the sample entered training In 1983 or 1984 
and had completed training by August 1984. The vast majority of the students were males; 
however, female students were included In proportion to their overall presence In the Air Force. 
Within the total sample, data for 5,091 students were to be used for model development, and data 
for 890 students were to be used In the cross-validation. There were no marked differences In 
the demographic characteristics of the student samples used for the model development and 
cross-validation. 



Preliminary Analyses 

Once the data had been collected and coded, preliminary data analyses were conducted. In 
these analyses, It was assumed that all course content variables could be applied to all students 
assigned to each course. Further, It was assumed that students with missing data for some 
variable would be omitted from only those analyses which Involved that variable. Given these 
assumptions, the overall mean and standard deviation for each course content, student Input, and 
training outcome variable were obtained In the total sample, the model-development sample, and 
the cross-validation sample. Subsequently, the mean and standard deviation of each student 
input, course content, and training outcome variable were obtained for each course. 
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Table 2 . Demographic Information by Course and Training Center 



Soecialtv 




Ant"f tiufp 


1 UIO 1 






Year 


code 


Smj>le a 


miniiHin 

IN 9 119 HTMH 


student ^ 


Male<; 




train 


42331 


M 


M30 


93 


qn 


0 


00 


42333 


M 


M35 


64 


58 


6 

u 


A3 
oo 


42735 


M 


M35 


73 


66 


7 


83 


44330 


M 


M35 


73 


73 
/ 0 


u 


00 


47231 


M 


M30 


62 


62 


0 


83 


57130 


M 


G40 


130 


IOU 


u 


OO 
OO 


42330 


V 


E40 


4Q 


44 




P. "3 
00 


42632 


V 


M30 


18fi 

IOU 


1 0£ 


£H 


QO 
OO 








730 


OOO 


HO 




11430 


M 


M35 


3P, 
oo 


37 
0/ 


1 


00 


36231 


M 


E45 


R4 


0 1 


q 
0 


oo 
00 


36131 


M 


M35 


4fi 


4fi 


U 


oo 
o3 


43131 


M 


M3R 
noo 


3Q 


00 


1 


82 


54230 


M 


M40 


71 


DO 


0 


OA 
OH 


55230 


M 


M3R 
noo 


lni 
1 U 1 


IUU 


1 


OA 

84 


67232 


M 


nUJ 




67 


oc 
00 


83 


90230 


M 


G45 


261 




OO 


00 


90232 


M 


G45 


95 


74 


21 


83 


90630 


M 




13fi 

1 OD 




AA 


OO 

00 


91330 


M 


G45 


57 


40 


17 


83 


98130 


M 


G45 


1 07 


no 

yo 


*t 1 


oo 
00 


67231 


V 


A65 


10? 


DO 


0/ 


QO 
00 


92430 


v 


645 


1 OH 


196 
Icq 


00 


oo 
oo 








1399 


1100 


299 




81130 


M 


G30 


926 


926 


0 


83 


81132 


M 


G35 


304 


277 


27 


83 








1230 


1203 


27 





Chtnute technical training center 
^Aircraft Environmental Systems Mechanic 
^Aircraft Fuel System Mechanic 
' Airframe Repair Specialist 
\ Mlssle Maintenance Spec 4 *l1st 

Special Vehicle Mechanic 

fire Protection Specialist 

Aircraft Electrical Systems Specialist 

Jet Engine Mechanic 

Total Students 

Sheppard technical training center 
"Aircraft Loadmaster 
Telephone Switching Specialist 
Cable Splicing Specialist 
Tactical Aircraft Maintenance Specialist 
Electrician 
/Carpentry Specialist 
financial Services Specialist 
r ;Medical Services Specialist 
Surgical Services Specialist 
Medical Administrative Specialist 
"Physical Therapy Specialist 
Dental Assistant Specialist 
\ Financial Management Specialist 
T Medical Laboratory Specialist 
; v Total Students 

s 

^Lackland security specialist center 
) Security Specialist 
[ Law Enforcement Specialist 

-Total Students 
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Table 2. (Concluded) 



Specialty 




Aptitude 


Total 






Year of 






Sample 3 


minimum 


students 


Males 


Females 


training 


'l^flfrfts ♦■prhn'iral tPalninn renter 
















srPnnt - 4 mimic Dhntri Prnrocci nn Inprial ict" 


P3330 


M 


G45 


90 


73 


1/ 


83 


viTnct inline nt at inn Mpchanir 


31633 


M 


E65 


42 


40 


2 


82 


Ayionic Sensor Systems Specialist 


32232 


M 


E70 


34 


33 


1 


G3 


precision Measuring Equipment Specialist 


32430 


M 


E65 


136 


129 


7 


83 


- AcrU->JJaU.c UiOUNU CW}U IJ-nlPcNl. OJJCU 1 O 1 1 i> L 


32630 


M 


E75 


CA 
54 


<tc 


12 


83 






M 


E65 


20 


20 


0 


83 


Attark Pnntrnl ^vctemc ^nerialict 


32636 


M 


E65 


63 


61 


2 


83 


Munitions Systems Sneriali<;t 


46130 


M 


M45 


237 


225 


12 


83 


Materiel Facilities Soecialist 


64531 


M 


G30 


158 


142 


16 


83 


Armament ^vctpmc ^neriali^t 


46230 


V 


M45 


68 


64 


4 


83 


- Total Students 








902 


829 


73 




ICfc#sler techniral trainina renter 
















Cnmanri and Pnivtrnl *»ner i a 1 1 ct 


27430 


M 


G50 


21 


17 


4 


83 


vtiiiucudiiu uumnun i cat i on cmJiynicNL djjcu id list 




M 


E65 


143 


139 


4 


83 


electronic uun^jULcr opcuiaiist 




M 


E65 


92 


90 


2 


83 


, i c icvonmun mat i oris control opcuidiist 




M 


E65 


81 


75 


6 


84 


Ai¥*hnr*no Uarn^nn ^vcf mac ^npri al i ct 
nil uui lie nai n i ny ojrouoiio ojjcu i a i i ol 


32832 


M 


E65 


74 


74 


0 


83 


Plprtrnn'ir Warfare ^vctpmc. ^nerialict 


3P833 


M 


E65 


117 


117 


0 


83 


-computer uperator 


Dl lOU 


M 


G40 


152 


202 


50 


83 


Administration Specialist 


70230 


M 


A35 


417 


301 


116 


83 


Personnel Specialist 


73230 


M 


A50 


184 


111 


73 


83 


.(Personal Affairs Specialist 


73231 


M 


A50 


68 


40 


28 


83 


Ground Radio Operator 


29333 


V 


A50 


113 


76 


37 


83 


: Aircraft Warning Radar Specialist 


30332 


V 


E75 


85 


82 


3 


83 


Navigation Systems Specialist 


32831 


V 


E65 


77 


74 


3 


83 


Computer Programming Specialist 


51131 


V 


G55 


46 


41 


5 


83 


Total Students 








1720 


1389 


331 





*M - Course included 1n model-development sample; V - Course included 1n cross-validation sample. 
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After these preliminary descriptive analyses had been conducted, an attempt was made to 
establish the simple blvarlate relationships among all variables. This was accomplished by 
intercorrelatlng scores on all of the variables being considered for Inclusion 1n the model In 
the total sample, and then obtaining separate estimates of these relationships within the 
model-development and cross-validation samples. This basic descriptive Information provided a 
framework for reviewing the initial conceptual model, and reevaluating initial conceptualizations 
of the relationships among the variables. This revision and refinement of the initial 
hypothetical model was a highly intuitive process carried out in a series of discissions with 
knowledgeable Air Force personnel. 



Priwary Analyses 

Once an acceptable revision of the initial hypothetical model had been formulated, model 
development war, undertaken. This was a relatively straightforward process in which the LISREL V 
program for the estimation of linear structural models was applied to the correlations obtained 
within the model-development sample (Jcraskog & Sorbom, 1980). All analyses were carried out 
using an unweighted least squares procedure. Application of this strategy offered certain 
advantages 1n terms of cost and the robustness of parameter estimates, although it did prohibit 
obtaining estimates of the standard error of the path coefficients. 

The LISREL V program operates by taking a series of specified causal relationships among the 
variables and generating parameter estimates that will maximize the anount of covariation in the 
observed data which can be accounted for by the hypothetical model. As well as allowing the user 
to specify causal paths between exogenous (i.e., independent) variables and endogenous (I.e., 
dependent) variables, LISREL V allows specification of causal paths among endogenous variables. 
In estimating these parameters, the LISREL V program can take into account correlations among 
observed or latent exogenous variables and endogenous variables, as well as correlated error 
terms among exogenous variables and correlated error terms among endogenous variables. In all 
analyses, correlation matrices were assumed to provide legitmate estimates to be included in the 
model. 

Once these decisions had been made, It was possible to run the model through the LISREL V 
program. Initially, the goodness-of-f it test and the amount of residual variance unaccounted for 
by the model were examined. The swltlple correlation coefficients (Rs) for the prediction of all 
training outcomes given the hypothesized causal relationships specified by the model were 
obtained, along with regression coefficients and standardized path coefficients specified by the 
causal relationships among the exogenous and endogenous variables and among the endogenous 
variables alone. Estimates of the total effect of each variable on all relevant dependent 
variables were also generated along with the requisite correlation matrices. It should be noted 
that these regression weights and path coefficients are not Identical to Rs but rather, represent 
weights for prediction. 

Once this model had been generated and the relevant data examined, an attempt was made to 
evaluate the predictive efficiency of the model. The regression coefficients specified in the 
foregoing analyses were used to predict training outcomes In the nine cross-validation courses, 
given knowledge of the course content variables for each course end average values for student 
input variables. The agreement between predicted training outcomes and the observed training 
outcomes for each of the nine courses then served as the basis for evaluating the predictive 
utility of the model. On the whole, the analyses described above seemed adequate for the 
development of an empirical model for use In predicting the impact of aptitude requirement 
adjustments on Air Force Initial-skills training. 
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III. RESULTS 



Descriptive Analyses 

Table 3 presents means and standard deviations of scores on all of the student Input, course 
content, and training outcome variables for the total sample, the model -development sample, and 
the cross-validation sample. These data were derived by aggregating over students or courses 
within each sample. The actual value for each student Input, course content, and training 
outcome variable for each technical training course are provided in Appendices E, F, and G. With 
regard to student Inputs, it should be noted that ASVAB scores for all students are based on 1944 
norms. This fact Is Important for score Interpretation because In October 1984, soon after data 
had been collected for this effort, a new set of norms based on a 1980 youth population began to 
be used for the derivaticr. of ASVAB scores (Ree, Valentine, & Earles, 1985). Comparisons between 
the mean scores on the ASVAB aptitude composites for each sample and historical ASVAB data 
Indicated that sample means are typical of those found in Air Force enlistee populations. 
Examination of Table 3 indicates that most trainees were approximately 20 years old and had 
completed high school. Additionally, they had taken one or two courses In high school that were 
considered to be desirable prerequisites for the specialties to which they were assigned. The 
Interest measures Indicate that roughly half of the trainees were guaranteed a particular job 
specialty and that most trainees received one of their preferred specialties. Typically, these 
trainees were able to read at an eleventh grade level, and had evidenced some academic 
achievement motivation with respect to the courses they took In high school. 

On the whole, the means and the standard deviations of the various student Input variables 
were remarkably similar for the model-development and cross-validation samples. However, it 
appears that the cross-validation sample consisted of students of somewhat higher quality. This 
was evidenced by consistently higher average scores on the ASVAB aptitude composites In the 
cross-validation sample. This result appears to Indicate that the cross-validation sample was 
composed of courses associated with higher aptitude requirement mlnlmums. 

In reviewing the results obtained for the course content variables, a number of salient 
points should be noted. First, the resident technical training courses sampled were about 400 
hours 1n length and, on the average, were divided Into 49 units of Instruction (diversity). 
Roughly half of the courses operated on an 8-hour Instructional day, whereas the other half 
operated on a 6-plus-2-hour instructional day. Generally, one Instructor was available for every 
nine students. Instructors tended to have roughly 2 1/2 years of training experience, and the 
quality of the Instruction they provided was judged by instructor supervisors as being between 
good and satisfactory. Slightly less than half of the the total training time was devoted to 
hands-on Instruction, while Instructional aids were used on at least a dally basis and feedback 
in one form or another was provided once every 3 hours. On the average, training programs 
devoted 8.5 hours to each unit of instruction, and operated under an expected attrition rate of 
9%. Of the courses selected for study, slightly less than half were associated with specialties 
having selective reenlistment bonuses. On the whole, most courses made only moderate demands 
with regard to the mastery of abstract principles although there were substantial differences in 
this variable across courses. The average reading grade level of training materials was 10.4, 
Indicating that the material was quite appropriate for use with students having an 11.4 reading 
grade level. However, it should be recognized that this conclusion pertains only to training 
materials produced by ATC. Training materials produced by private vendors and technical orders 
were not Included In the analysis. The average learning difficulty of the tasks performed in 
specialties associated with the courses sampled was typical of that found for most AFSs. It 
should be noted that the average yearly student flow for the courses was quite high in the total 
and model-development samples relative to the cross-validation sample. This result 1s due to the 
inclusion cf a few high-flow courses, such as Security Specialist (AFS 81130), In the 
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Table 3. Means and Standard Deviations of Variables 



Model-development Cross-validation 
Total sample sample sample 
(N » 5,981) (N « S.091) (N ; 890) 



Student Input variables 3 


X 


SD 


X 


SD 


_ 

X 


SD 


Mechanical 


55.13 


22*40 


54.40 


22.30 


59.30 


22.60 


Administrative 


63.16 


19.50 


62.70 


19.50 


65.80 


19.70 


General 


65.28 


16.90 


64.70 


16.80 


68.60 


17.30 


Electronics 


63.46 


18.50 


62.70 


18.50 


67.80 


18.00 


Reading Level 


11.43 


1.00 


11.40 


1.00 


11.60 


1.00 


Academic Motivation 


38.94 


13.40 


38.40 


13.30 


42.00 


13.50 


Simple Interest 


1.41 


•49 


1.40 


.49 


1.50 


.50 


Preference Interest 


4.42 


•86 


4.40 


.87 


4.50 


.82 


Educational Level 


2.12 


•45 


2.10 


.42 


2.20 


.56 


Educational Preparation 


1.55 


.94 


1.60 


.87 


1.90 


1.10 


Age 


20.07 


2.20 


20.00 


2.10 


20.50 


2.30 


Course content variables^ 














Course Length 


420.73 


309.30 


386.30 


302.20 


569.90 


297.70 


Day Length 


.49 


.49 


.46 


.49 


.62 


.48 


Student-Faculty Ratio 


9.03 


4.80 


9.20 


4.90 


8.30 


3.90 


Instructor Experience 


33.43 


14.70 


29.70 


12.90 


49.60 


13.60 


Instructor Quality 


2.48 


.16 


2.50 


.16 


2.40 


.13 


Aids In Use 


.27 


• 10 


.27 


.10 


.28 


.13 


Hands-On Instruction 


.40 


.13 


.41 


.13 


.37 


10 


Amount of Feedback 


•34 


.12 


•35 


.13 


.28 


.06 


Amount of Practice 


8.49 


3.10 


8.60 


3.00 


8.00 


3.80 


Reenllstment Bonus 


.46 


.49 


.42 


.49 


.61 


.48 


Yearly Student Flow 


1852.33 


2662.50 


2204.90 


2782.00 


324.50 


118.00 


Occupational Difficulty 


99.36 


22.10 


93.40 


18.80 


125.20 


23.30 


Reading Difficulty 


10.98 


.63 


11.00 


.67 


10.90 


.30 


Abstract Knowledge 


2.39 


.98 


2.20 


.91 


3.20 


.93 


Expected Attrition 


.10 


.03 


.09 


. .03 


.13 


.02 


Diversity 


49.06 


43.30 


49.90 


41.10 


45.40 


45.20 


Training outcome variables 3 














Quality of Student Performance 


85.27 


7.60 


85.00 


7.70 


86.80 


6.90 


SIA Time 


6.54 


15.40 


5.90 


15.20 


10.20 


15.80 


Academic Counseling 


1.43 


3.40 


1.40 


3.40 


1.60 


3.40 


Nonacademic Counseling 


.17 


1.50 


.19 


1.60 


.08 


.86 


Retraining 


11.12 


51.50 


8.80 


42.70 


24.40 


84.50 


Academic Attrition 


.03 


.16 


•02 


.15 


.05 


.22 


Nonacademic Attrition 


•005 


• 06 


•004 


.06 


.011 


.03 



Statistics based on subject samples (Indicated at top of page). 



D Stat1st1cs based on course samples (Total Sample = 48 courses, Model -Development Sample = 
39 courses, Cross-Validation Sample s 9 courses). 
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model-development sample but not In the cross-validation sample. This Is evidenced by the 
sizable standard deviation associated with yearly student flow In the total and model-development 
samples but not the cross-validation sample. 

In examining the similarities and differences among the means and standard deviations of the 
course content variables across the model-development and cross-validation samples, another 
Interesting finding emerged. It was found that course length, expected attrition rates, abstract 
knowledge requirements, and occupational difficulty were all higher In the cross-validation 
sample than In the model-development sample. This confirmed the observation made with respect xo 
the student Input variables that, on the whole, the cross-validation courses are more difficult 
than the model-development courses. 

Having reviewed the descriptive characteristics of the student Input and course content 
variables, the results obtained for the training outcome variables will now be considered. It Is 
apparent In Table 3 that the Incidence of both academic and nonacademic attrition In the various 
technical training courses Is extremely low. In the total sample, only 167 academic attritions 
and 24 nonacademic attritions occurred. It should be recognized that the Infrequency of these 
outcomes makes prediction much more difficult. Academic and nonacademic counseling occurred far 
more frequently than did attrition. The average trainee could be expected to receive one and 
one-half academic counseling sessions, whereas one student In six could expect to receive 
nonacademic counseling. Given these academic and nonacademic attrition and counseling rates, it 
seems reasonable to conclude that academic concerns are far more likely to result In negative 
training outcomes than are nonacademic concerns such as motivation or attitude. Finally, it 
should be noted that the relatively large standard deviation associated with these variables nwy 
be attributed to the fact that only a few students received counseling or were eliminated from 
training. 

Overall, the quality of student performance In the various courses was adequate. Given a 
range of passing grades of 70 to 100, most students obtained a grade of 85. SIA appeared to be 
employed frequently In remediating errors, such that the typical student accrued an additional 
6.5 hours of one-on-one Instruction. However, the magnitude of the standard deviation Indicates 
that students In some courses received substantially more SIA than did others,, Retraining hours, 
while quite high at a mean of 11.1 hours In the total sample, also appear to be associated with a 
few students who received substantial amounts of retraining. This conclusion Is supported by the 
sizable standard deviation for this training outcome. Finally, It should be noted that the 
higher average attrition rates, retraining time, and SIA time observed In the cross-validation 
sample relative to the model-development sample suggest that the courses Included In the 
cross-validation sample were somewhat more demanding than those found In the model -development 
sample. 



Correlational Data 

Revision and extension of the Initial conceptual model was based on Information obtained 
during the Interviews and on statistical relationships found for all pairs of variables In the 
model-development sample. Consequently, a discussion of these relationships Is In order. 
Blvarlate correlations derived on the basis of data obtained from the model-development sample 
are presented In Table 4. Correlations derived on the basis of data obtained from the total 
sample and the cross-validation sample are presented in Appendices H and I. 

Turning first to the Interrelationships among the training outcome variables, the assessed 
quality of academic performance exhibited a pattern of negative relationships with all other 
training outcomes, as would be expected given the Impact of performance on decisions related to 
counseling, retraining, and attrition. The strongest relationships with assessed quality were 
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Table 4 , Intercorrelatlons of Student Input, Course Content, and 
Training Outcome Variables In the Model-Development Sample 
(5,091 students; 39 courses) 



Var1ab1es a Variables 







SI-1 


SI-2 


SI-3 


SI-4 


SI-5 


SI-6 


SI-7 


SI-8 


SI-9 


SI-10 


SI-11 


SI-12 CC-1 


CC-2 


CC-3 


CC-4 


CC-5 


CC-6 


SI-1 


Aptitude Selector 


1.00 


.47 


.48 


.75 


.64 


.53 


.35 


-.21 


-.15 


.18 


.08 


.18 


.25 


-.26 


-.14 


-.04 


.09 


-.13 


SI~2 


Mechanical 




1.00 


.26 


.47 


.66 


.42 


.25 


-.05 


-.02 


.09 


.03 


.15 


.30 


-.08 


-.23 


-.18 


.15 


.06 


SI-3 


Administrative 






1.00 


.44 


.27 


.35 


.26 


-.07 


-.06 


.15 


.06 


.10 


.06 


-.16 


-.01 


.09 


.01 


-.15 


SI-4 


General 








1.00 


.75 


.67 


.35 


-.13 


-.06 


.19 


.11 


.18 


.23 


-.14 


-.07 


.00 


.02 


-.18 


SI -5 


Electronics 










1.00 


.52 


.40 


-.10 


-.03 


.17 


.08 


.16 


.34 


-.16 


-.18 


-.08 


.10 


-.06 


SI-6 


Reading Level 












1.00 


.25 


-.08 


-.02 


.16 


.10 


.19 


.18 


-.10 


-.06 


-.02 


.02 


-.14 


SI-7 


Academic Motivation 














1.00 


-.08 


-.06 


.29 


.22 


.24 


.17 


-.15 


-.04 


.00 


.07 


-.05 


SI-8 


Simple Interest 
















1.00 


.71 


-.07 


-.05 


-.10 


-.02 


.17 


.04 


-.10 


-.05 


.08 


SI-9 


Preference Interest 


















1.00 


-.07 


-.02 


-.04 


.00 


.13 


.00 


-.11 


-.01 


.04 


SI-10 


Educational Level 




















1.00 


.06 


.43 


.07 


-.08 


-.03 


.00 


.03 


-.07 


SI-1 1 


Educational Prep. 






















1.00 


.06 


.20 


.42 


.05 


.05 


-.07 


-.07 


SI-12 


Age 
























1.00 


.13 


-.13 


-.07 


.00 


.06 


-.08 


CC-1 


Course Length 


























1.00 


-.37 


-.42 


-.19 


.48 


.06 


CC-2 


Day Length 




























1.00 


.42 


.00 


-.24 


-.01 


CC-3 


Student-Faculty Ratio 






























1.00 


.48 


-.40 


-.39 


CC-4 


Instructor Quality 
































1.00 


.02 


.11 


CC-5 


Instructor Experience 


































1.00 


.00 


CC-6 


Aids In Use 




































1.00 



CC-7 Hands-On Practice 

CC-8 Amount of Feedback 

CC-9 Practice 

CC-10 Reenlistment Bonus 

CC-11 Student Flow 

CC-12 Occupational Difficulty 

CC-13 Reading Difficulty 

CC-14 Diversity 

CC-15 Abstract Knowledge 

CC-16 Expected Attrition 

T0-1 Quality of Performance 

T0-2 SIA Time 

TO- 3 Academic Counseling 

T0-4 Nonacademic Counseling 

TO-5 Retraining Time 

TO-6 Academic Attrition 

T0-7 Nonacademic Attrition 
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Table 4 , (Concluded) 



Variables* Variables 
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CC-7 


CC-8 


CC-9 


CC-10 CC-11 


CC-12 CC-13 CC-14 


CC-15 CC-16 T0-1 


TO-2 


TO-3 


TO-4 


TO-5 


TO-6 


T0-7 


SI-1 


Aptitude Selector 


.21 


.05 


-.13 


-.01 


-.25 


.03 


.27 


.27 


.21 


.19 


.37 


-.03 


-.15 


-.07 


-.09 


-.08 


-.04 


SI-2 


Mechanical 


-.09 


-.07 


-.04 


.19 


-.18 


.34 


.06 


.30 


.30 


.19 


.25 


.03 


-.09 


-.03 


-.04 


-.05 


-.02 


SI -3 


Administrative 


-.14 


.04 


.07 


-.07 


-.13 


-.12 


.18 


.07 


.02 


.04 


.24 


-.07 


-.13 


-.06 


-.09 


-.06 


-.03 


SI-4 


General 


-.23 


-.01 


-.11 


.05 


-.18 


.05 


.13 


.26 


.26 


.12 


.33 


-.04 


-.13 


-.04 


-.08 


-.07 


-.02 


SI-5 


Electronics 


-.20 


-.03 


-.12 


.14 


-.17 


.23 


.16 


.37 


.35 


.21 


.32 


.00 


-.10 


-.05 


-.06 


-.06 


-.04 


SI-6 


Reading Level 


-.18 


-.04 


-.06 


.04 


-.13 


.05 


.10 


.19 


.19 


.12 


.32 


-.05 


-.15 


-.02 


-.08 


-.07 


-.01 


SI-7 


Academic Motivation 


-.1 


1.00 


-.08 


.03 


-.12 


.07 


.14 


.20 


.16 


.13 


.21 


-.02 


-.07 


-.05 


-.05 


-.05 


-.04 


SI-8 


Simple Interest 


.08 


-.16 


.16 


.17 


.29 


.02 


-.05 


-.08 


-.11 


-.01 


-.11 


-.04 


.02 


.02 


.01 


.02 


.00 


SI-9 


Preference Interest 


.00 


-.13 


.12 


.12 


.17 


.04 


-.02 


-.03 


-.02 


-.05 


-.08 


-.06 


.00 


.0 


1.00 


.02 


.00 


SI-10 


Educational Level 


-.09 


.00 


-.03 


.03 


-.09 


-.02 


.08 


.06 


.05 


.05 


.15 


-.02 


-.04 


-.03 


-.01 


-.04 


-.03 


SI-1 1 


Educational Prep. 


.05 


.15 


-.29 


-.20 


-.18 


-.12 


-.03 


.02 


.34 


.00 


.02 


.12 


-.04 


.00 


-.05 


-.03 


-.01 


SI-12 


Age 


-.10 


.00 


-.08 


-.02 


-.16 


.02 


.14 


.14 


.13 


.11 


.16 


.01 


-.02 


-.03 


.00 


.02 


-.02 


CC-1 


Course Length 


-.25 


-.11 


-.03 


.19 


-.28 


.56 


.90 


.87 


.58 


-.66 


.19 


.14 


.12 


-.03 


.07 


.01 


-.02 


CC-2 


Day Length 


.04 


-.02 


.27 


.13 


.46 


-.10 


-.52 


-.39 


-.23 


-.36 


-.30 


-.05 


-.03 


.07 


.05 


.00 


.05 


CC-3 


Student-Faculty Ratio 


.0; 


.07 


.01 


-.18 


.26 


-.55 


-.30 


-.36 


-.20 


-.29 


-.21 


-.0 


1.00 


.04 


.07 


.00 


.02 


CC-4 


Instructor Quality 


.0/ 


.57 


-.37 


-.57 


-.26 


-.45 


.16 


-.10 


.06 


-.08 


.10 


.02 


-.04 


-.03 


-.04 


.00 


.00 


CC-5 


Instructor Experience 


-.24 


.08 


-.18 


-.08 


-.36 


.48 


.32 


.47 


.26 


.15 


.24 


.05 


.04 


-.05 


.03 


.00 


-.01 


CC-6 


Aids In Use 


.56 


.00 


-.02 


.34 


.33 


.23 


.06 


.10 


.00 


.02 


.01 


.11 


.04 


.00 


.07 


.02 


-.04 


CC-7 


Hands-On Practice 


1.00 


-.08 


.12 


.14 


.46 


-.12 


.07 


-.27 


-.13 


.09 


-.08 


.12 


.01 


.02 


,03 


.02 


-.01 


CC-8 


Amount of Feedback 




1.00 


-.40 


-.41 


-.40 


-.16 


.09 


.09 


-.06 


-.22 


.20 


.00 


-.04 


-.03 


.00 


.00 


-.01 


CC-9 


Practice 






1.00 


.30 


.56 


.09 


-.30 


-.37 


-.34 


.17 


-.29 


-.15 


.02 


.07 


-.04 


.00 


.06 


CC-10 


Reenllstment Bonus 








1.00 


.52 


.36 


-.06 


.13 


.02 


.14 


-.19 


.00 


.10 


-.06 


.04 


.00 


.00 


CC-1 1 


Student Flow 










1.00 


-.12 


-.26 


-.42 


-.45 


-.16 


-.32 


-.20 


.00 


.08 


-.0 


1.00 


.01 


CC-1 2 


Occupational Difficulty 










1.00 


.06 


.54 


.44 


.46 


.05 


.13 


.08 


.00 


.06 


.0 


1.00 


CC-13 


Reading Difficulty 














1.00 


.43 


.33 


.48 


.29 


.04 


-.01 


-.06 


.02 


.01 


-.05 


CC-1 4 


Diversity 
















1.00 


.63 


.51 


.25 


.16 


.00 


-.04 


.09 


.02 


-.03 


CC-1 5 


Abstract Knowledge 


















1.00 


.49 


.11 


.32 


.09 


-.03 


.03 


.03 


-.03 




Expected Attrition 




















1.00 


.07 


.16 


.07 


-.01 


.03 


.0 


1.00 


TO-1 


Quality of Performance 






















1.00 


-.09 


-.27 


-.19 


-.14 


-.17 


-.07 


TO-2 


SIA Time 
























1.00 


.23 


.00 


.10 


.06 


-.01 


TO-3 


Academic Counseling 


























1.00 


.22 


.49 


.50 


.05 


TO-4 


Nonacademlc Counseling 




























1.00 


.24 


.19 


.49 


TO-5 


Retraining Time 






























1.00 


.63 


.14 


TO-6 


Academic Attrition 
































1.00 


.00 


TO-7 


Nonacademlc Attrition 
































1.00 
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training outcome variables are denoted by the abbreviation TO* 



produced by academic and nonacademlc counseling, respectively. Academic counseling, in turn, 
yielded strong positive relationships with academic attrition and retraining time, whereas 
academic attrition and retraining time In turn displayed a strong positive relationship. This 
result was expected because academic counseling necessarily precedes attrition or retraining 
decisions. Similarly, a strong positive relationship was obtained between nonacademlc counseling 
and nonacademlc attrition, and a moderate positive relationship was observed between academic and 
nonacademlc counseling. As might be expected on the basis of the coding procedures used, 
academic and nonacademlc attrition were not correlated. A surprising finding was that SIA time 
was not strongly related to any other training outcome variable except academic counseling. The 
weak relationships found for SIA time and other training outcomes may be attributed to the fact 
that SIA is used as a remediation technique and is related to other outcomes only to the extent 
that It reveals underlying problems which cause students to be referred for academic counseling. 

The most striking relationships observed for training outcome variables was that final 
outcomes like counseling (nonacademlc and academic) and attrition (nonacademlc and academic) 
exhibited weak relationships with the antecedent student Input and course content variables, 
whereas Immediate outcomes such as assessed quality and SIA time displayed some moderate 
relationships with the antecedent student Input and course content variables. This suggests that 
any effect student input and course content variables have on final outcome variables (i.e., 
counseling and attrition) is moderated by the Immediate outcomes, SIA time and assessed quality 
of performance. Assessed quality of performance exhibited only moderate positive relationships 
with the ASVAB aptitude composites. However, such relationships were expected because 
correlational analyses were conducted by combining courses of widely varying difficulty. It was 
also found that age, academic motivation, and educational level exhibited a pattern of positive 
relationships with assessed quality of performance. As expected, motivation, maturity, and 
educational level enhance training performance. The weak negative relationships between assessed 
quality and the two Interest measures are counter-intuitive and may reflect inadequacies in these 
measures. The absence of a relationship between educational preparation and assessed quality may 
be attributed to the moderating Influence of other student Input or course content variables. 

Among the course content variables, both Instructor quality and Instructor experience were 
positively related to assessed quality of performance. This finding supports the notion that 
effective teaching enhances student performance. It was found that academic day length and 
student-faculty ratio were negatively related to assessed quality of performance. These results 
appear to support suggestions by interviewees that an 8-hour academic day Is too much for 
students to handle, and that student performance decreases when Instructors are burdened with too 
large a class to devote sufficient time to Individual students. Amount of feedback was 
positively related to assessed quality of performance, supporting the Intuitive notion that 
frequent feedback Increases training performance. Amount of practice exhibited a negative 
relationship with assessed quality of performance. The reason for this unexpected result Is not 
clear, but it could be due to the moderating effect of other variables. Student flow exhibited a 
negative relationship with assessed quality of performance. This result was expected because 
high-flow courses are often associated with specialties having low aptitude requirements and 
because high-flow courses often have poor student-faculty ratios. It was found that course 
length exhibited a moderate positive relationship with assessed quality of performance. This 
finding supports the notion that training performance Improves when students are given more time 
to learn. It was surprising to find that course diversity, abstract knowledge, and reading 
difficulty exhibited a pattern of positive relationships with assessed quality of performance. 
However, this result might be attributed to the tendency for more demanding courses to receive 
higher aptitude students. Such a hypothesis Is supported by the correlations between these sane 
variables and the ASVAB aptitude composites. 
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Few strong correlations were obtained between SIA time and student Input and course content 
variables. It was found that SIA time was positively related to course length, as might be 
expected since longer courses would provide greater opportunity for SIA. It was also found that 
SIA time was nt%c ;*ly related to student flow, as was expected given the fact that the 
Instructional demands created by a large class would tend to decrease the likelihood of SIA. 
Abstract knowledge, occupational difficulty, diversity, and expected attrition all yielded 
positive relationships with SIA time. These relationships may Indicate that SIA time 1s likely 
to be higher 1n more difficult training courses. Use of instructional aids and hands-on practice 
were also found to exhibit positive relationships with SIA time, perhaps because the use of aids 
and hands-on performance m*y provide an overt signal to Instructors that could alert them to the 
need for SIA. Finally, it should be noted that SIA time was effectively unrelated to student 
Input variables, suggesting that the effects of student attributes on SIA time m*y be moderated 
by the assessed quality of performance or course content variables. 

In examining the Interrelationships among student Input variables, a number of Interesting 
findings emerged. First, scores on the ASVAB aptitude composites displayed a pattern of positive 
Interrelationships, which Is due to the fact that some aptitude composites share cooroon 
subtests. Similarly, the aptitude measures displayed positive relationships with reading level 
as measured by the AFRAT. Scores on all of the aptitude composites displayed a pattern of 
positive relationships with educatl9nal level, educational preparation, academic motivation, and 
age. Although these relationships were expected, given the tendency of aptitude to facilitate 
academic motivation, encourage education, and maximize time spent 1n school, it should be noted 
that the strength of these relationships did not Indicate a one-to-one correspondence. Finally, 
It sliould be noted that although a strong positive relationship was observed between the two 
interest measures, both measures exhibited a pattern of weak negative relationships with measures 
of aptitude, academic motivation, educational level, educational preparation, and age. This 
unexpected finding may be Interpreted as a side effect of the personnel allocation system wherein 
the more talented Individuals are most 1n demand, and therefore may be more likely to be given 
assignments they did not expressly request. 

In examining the correlations between student Input and course content variables, only 
trivial relationships were found for the most part. However, these two classes of variables are 
not Independent within the context of Initial-skills training. Scores on the various aptitude 
measures and students 1 reading levels exhibited a pattern of positive relationships with course 
length, course reading difficulty, abstract knowledge requirements, diversity, arid expected 
attrition rates. This pattern of results would seem to Indicate that high-ability Individuals 
are more likely to be assigned to specialties having demanding training programs. 

Upon examination of the Interrelationships among the various course content variables, a 
complex pattern of relationships emerged. It was found that occupational difficulty, course 
reading difficulty, diversity, abstract knowledge requirements, course length, and expected 
attrition displayed the strongest Interrelationships of all course content variables. Due to the 
similarity of the variables, 1t seems reasonable to hypothesize that they represent a factor 
which reflects course subject-matter difficulty. 

One course content variable that exhibited a conspicuous pattern of strong-to-moderate 
relationships was yearly student flow. Student flow was found to yield negative relationships 
with occupational difficulty as well as course reading difficulty, diversity, abstract knowledge, 
course length, and expected attrition. These results appear to Indicate that relatively fewer 
students enter the more difficult training courses. The negative relationships between student 
flow and both Instructor quality and Instructor experience may be attributed to the difficulty of 
obtaining sufficient numbers of experienced, quality Instructors when the class size Is routinely 
large. The positive relationship between student flow and student-faculty ratio probably 
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reflects the Influence of student flow on class size* The positive relationship between flow and 
academic day length suggests that the less difficult, high-flow courses would have lo.iger 
academic training days. The positive relationships obtained between flow and both Instructional 
aids and hands-on Instruction may be due to the tendency for Investment In training support to be 
economically more feasible when a large number of individuals are to be trained. Student flow 
was negatively related to feedback, suggesting that student evaluation may be less frequent when 
a large number of students must be trained. On the other hand, student flow was positively 
related to practice, Indicating that high-flow courses typically devote more training hours to 
each unit of instruction. Flow also exhibited a strong positive relationship with reenlistment 
bonus, as was expected given that manning needs underlie both selective ^enlistment bonuses and 
student flow. In summary, the large number of strong relationships between student flow and 
other course content variables Indicate that student flow plays a central role In the design of 
training courses. 

Although practice yielded a strong positive relationship with flow, it exhibited a 
substantial negative relationship with feedback. The negative relationship with feedback was 
expected In view of the fact that feedback In the form of progress checks, performance tests, or 
sampling written tests are required by Instructional unit. The shorter ihese units over the 
deration of the course, the greater the feedback. Practice was positively related to 
Instructional day length since a longer instructional day provides for more practice per unit of 
Instruction, it was found that Instructor quality and instructor experience were positively 
related to practice and negatively related to feedback, perhaps because with long Instructional 
units and limited feedback, there Is less need for experienced, quality instructors. Confirming 
this hypothesis, feedback was found to be strongly related to Instructor quality. It was also 
found that neither feedback nor practice produced a systematic pattern of relationships with 
course subject-matter difficulty Indices, such as the reading difficulty of course materials. 

Frequency of use of Instructional aids was negatively related to student-faculty ratio. This 
finding may be attributed to the fact that high student-faculty ratios may limit the amount of 
Instructor supervision that can be devoted to the use of aids. The use of aids was highly 
correlated with hands-on practice. This relationship was not surprising in view of the manner in 
which the instructional aids variable was defined and measured. 

No systematic relationship was found between Instructor quality and instructor experience; 
however, Instructor experience was related to the occupational difficulty and Indices of course 
subject-matter difficulty. Apparently, more experienced Instructors are assigned to the more 
difficult courses. Both Instructor quality and instructor experience were negatively related to 
student-faculty ratio. This result Indicate that difficult courses to which more experienced 
Instructors are assigned are more likely to have fewer students per class. r ie appropriateness 
of this interpretation is supported by the negative correlation between Instructor experience and 
Instructional day length which also appears to be Influenced by student flow. It should also be 
noted ,hat st^it-faculty ratio and day length were positively related, perhaps due to the 
influence of student flow. Finally, both student-faculty ratio and day length were Inversely 
related to occupational difficulty and Indices of course subject-matter difficulty, perhaps 
because longer instructional days and larger classes are less likely to be appropriate as the 
difficulty of the training material increases. 



Model Rcfinemer * 

Earlier in this report, a rather simple conceptual model of the relationship among student 
inputs, course content variables, and training outcomes was presented. However, until the 
variables within each of these categories had been specified and interrelated, it was difficult 
to specify the model in greater detail. Having examined these relationships, it became possible 
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to construct a more detailed conceptual model of the relationships among the student Inputs, 
course content variables, and training outcomes. Because this elaboration of the conceptual 
model Is a critical step In the approach being followed In the present effort, this refinement of 
the conceptual model In relation to the Intercorrelatlons among the student Input, course 
content, and training outcome variables will be presented In some detail. 

As noted earlier, academic counseling, nonacademlc counseling, retraining, academic 
attrition, and nonacademlc attrition consistently exhibited weak relationships with student Input 
and course content variables, whereas assessed quality of performance and SIA time exhibited some 
moderate relationship with student Input and course content variables. This observation 
suggested that the effect of student Input and course content variables on training outcomes must 
operate through the assessed quality of student performance and SIA time. Moreover, the observed 
correlations support the common-sense notion that assessed quality of performance has a direct 
causal Impact on both academic counseling and nonacademlc counseling, while SIA time has a direct 
causal Impact only on academic counseling. Further, It seems reasonable to assume that assessed 
quality of performance would have a direct causal Impact on SIA tlma, although It 1s not expected 
that this effect Is strong In view of the seak relationship between these variables. 

Gwen this conceptualization of the Immediate training outcomes, academic counseling and 
nonacademlc counseling must be viewed as moderating the relationship between assessed quality and 
distal training outcomes such as retraining and attrition. This seems well justified because 
Interview data Indicated that either academic or nonacademlc counseling necessarily precedes 
student elimination and retraining decisions. Thus, academic counseling was viewed as having a 
direct causal Impact on academic attrition and retraining while nonacademlc counseling was held 
to have a direct, causal Impact on nonacademlc attrition. Because the Interviewees also reported 
that frequent retraining usually led to academic attrition and because retraining and academic 
attrition were highly correlated, It was held that retraining would have a direct causal Impact 
on academic attrition. It was also assumed that academic outcomes such as retraining and 
academic attrition would have no direct causal effect on nonacademlc attrition. This assumption 
seems appropriate In view of the observed relationships and Interview data which Indicated that 
retraining was an academic action unrelated to nonacademlc Issues. However, It was held that 
academic counseling might have a direct causal Impact on nonacademlc counseling. This hypothesis 
was In part based on the observed pattern of relationships, and In part on the assumption that 
academic counseling might occasionally uncover nonacademlc problems. 

The preceding discussion specified hypotheses concerning the structure of the relationships 
cimong training outcomes. At this point, It Is appropriate to turn to the expected relationships 
between the student Input and training outcome variables. At the risk of Introducing 
specification error, It was decided that the two vocational Interest measures should not be 
Included In the refined model. This decision was based on the ambiguity pertaining to the 
validity of the Interest measures and the unexplained negative correlations between the Interest 
measures and assessed quality of performance. Once this decision had been made, the pattern of 
the observed relationships between student Inputs and training outcomes suggested that all of the 
remaining student Input variables, (e.g., aptitude, reading level, academic motivation, 
educational level, educational preparation, and age) should be assumed to have direct impacts on 
assessed quality of student performance. Based on the observed weak relationships between 
student Input variables and SIA, direct Impacts on SIA time were not assumed. Rather, It was 
held that any Impact student Inputs had on SIA time was moderated through assessed quality of 
student performance or the course content variables. 

Having specified the relationships among the student Input and the training outcome 
variables, attention will now turn ,o the relationships specified for the course content 
variables. In the preceding discussion, It was suggested that student flow and course 
subject-matter difficulty were prime determinants of the relationships observed among course 



content variables, such as student-faculty ratio and instructional day length. When this 
observation Is coupled with the fact that training must consider the learning difficulty of 
occupational tasks, 1t appears reasonable to assert that training is driven by the difficulty of 
occupational tasks, the difficulty of course subject matter, and manpower requirements. 

Although occupational difficulty coi 11 ^ be measured directly, manpower requirements and course 
subject-matter difficulty represented ••cent variables that were defined through observed 
variables. The high correlation found bf.^een student flow and reenlistment bonus suggested that 
manpower requirements should be defined by these two variables. Given the pattern of 
correlations discussed earlier, it seemed that course subject-matter difficulty could be defined 
by the reading difficulty of course materials, the diversity of course materials, abstract 
knowledge requirements, expected attrition rates, and course length. However, it was held that 
only the former four variables would be used to define subject-matter difficulty since course 
length 1s reasonably considered an outcome of course subject-matter difficulty. 

According to this conceptualization, course subject-matter difficulty, occupational 
difficulty, and manpower requirements were held to be the three fundanental or primal variables 
affecting training design. These variables were assumed to bs correlated with each other. Also, 
these three primal variables were held to determine lower-order course variables, such as 
instructional d*y length and student-faculty ratio. Thus, course length, day length, 
student-faculty ratio, Instructor quality, instructor experience, use of aids, hands-on practice, 
feedback, and practice were held to be determined by these three variables. However, based on 
the observed correlations, it was decided that instructional aids and hands-on practice should be 
combined into a latent variable representing Instructional support, whereas feedback and practice 
should be combined Into a latent variable representing feedback intensity. 

Feedback intensity, Instructional support, instructor experience, Instructor quality, 
student-faculty ratio, day length, and course length were all assumed to be capable of having a 
direct causal Impact on both the assessed quality of student performance and SIA time. The 
direct causal relationships with SIA time were postulated for these course variables, unlike the 
student inputs, because it was felt that course content variables such as course length or day 
length could affect the feasibility of SIA time regardless of performance. Further, it was 
assumed that independent of the effects of the various course variables on quality of 
performance, subject-matter difficulty, occupational difficulty, and manpower requirements might 
also have a direct causal impact on both assessed quality of performance and SIA time. 

A schematic representation of the refined model is presented in Figure 3. As may be seen, 
the student input variables are assumed to be intercorrelated and have a direct causal impact on 
the assessed quality of performance. Course subject-matter difficulty, occupational difficulty, 
and manpower requirements are assumed to have a direct causal impact on all other course 
variables as well as on SIA time and the assessed quality of performance. Finally, all 
lower-order course variables, such as day length or student-faculty ratio, are assumed to have a 
direct causal Impact on both the assessed quality of performance and SIA time. 

The causal paths above the course parameters reflect the hypothesized effects of the three 
r imal variables 1n order from left to right, while the causal paths below the course parameters 
reflect their effects on quality of performance and SIA time, respectively. Finally, the diagram 
shows that assessed quality of performance and SIA time determine all other training outcomes 
when moderated through academic and nonacademic counseling. 
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Figure 3 . Refined Conceptual Model of Variables. 
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Modeling Results 



When the refined model was analyzed by LISREL V, 1t was found to provide an excellent fit to 
the observed relationships among the student . input, course content, and training outcome 
variables within the model-development sample. Overall, it was found that the model yielded a 
goodness-of-f1t Index of .59, and a residual of .19. This goodness-of-f it Index indicates that 
the refined model provided an unusually close fit to the observed data. This was confirmed by 
the residual term, which indicated that only 19% of the variance 1n the relationships among the 
three classes of variables could not be accounted for by the hypothetical model. 

Evidence of the predictive power of the model may be obtained by considering the Rs generated 
by the model against each of the dependent variables. These data are presented in Table 5 and 
broken down with respect to whether the dependent variable was a training outcome measure or a 
predicted course variable. For the training outcome variables, the model yielded an R of .75 
against assessed quality of performance. Multiple correlation coefficients of .60 and .50 ware 
obtained against academic and nonacademic counseling, respectively; retraining time, academic 
attrition, and nonacademic attrition produced Rs of .76, .83, and .59, respectively. Overall, 
this pattern of results Indicates excellent prediction of training outcomes, with academic 
outcome variables being somewhat better predicted than nonacademic outcomes such as nonacademic 
counseling and nonacademic attrition. The weakest prediction of a training outcome variable was 
obtained for SIA time, which yielded an R of .35. Apparently, SIA time represents a somewhat 
unique variable among the training outcomes. 



Table 5 . Multiple Rs ard R*s for Training Outcome and 
Course Content Variables 





R 


r2 


Training Outcome Variables 






Assessed Quality of Performance 


.75 


.54 


SIA Time 


.35 


.12 


Academic Counsel ing 


.60 


.36 


Nonacademic Counseling 


;50 


.25 


Retraining Time 


* .76 


.59 


Academic Attrition 


.83 


.70 


Nonacademic Attrition 


.sg 


.35 


Predicted Course Variables 






Course Length 


.86 


.75 


Day Length 


.60 


.36 


Student-Faculty Ratio 


.62 


.38 


Instructor Quality 


.58 


.34 


Instructor Experience 


.60 


.36 


Instructional Support 


.54 


.30 


Feedback Intensity 


.68 


.47 



For the course variables that were held to be determined by course subject-matter difficulty, 
occupational difficulty, and manpower requirements, a similar pattern of results was obtained. 
These three variables yielded Rs against course length, Instructional day length, student-faculty 
ratio, Instructor quality, instructor experience, Instructional support, and feedback intensity 
of .86, .60, .62, .58, .60, .54, and .68, respectively. Overall, the magnitude of these Rs 
Indicates that an optimal combination of subject-matter difficulty, occupational difficulty, and 
manpower requirements did an excellent job of predicting the lower-order course variables. This 
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Is an Important finding because It suggests that It is fully appropriate to conceive of training 
as being driven by course subject-matter difficulty, the difficulty of occupational tasks, and 
manpower requirements. 

The foregoing results Indicated that the refined model was capable of accounting for the 
relationships observed among the student Input, course content, and training outcome variables, 
as well as yielding excellent prediction of training outcomes. Given this observation, It would 
seem appropriate to examine the nature and magnitude of the standardized path coefficients 
generated by the lISREL V program In optimizing regeneration of the varlance-covarlance matrix. 
The standardized path coefficients generated by the LISREL V program are presented In Figure 4. 
A standardized path coefficient represents the unique effects of an Independent variable on a 
dependent variable when both are expressed 1n standard score form. It can be Interpreted as the 
magnitude of change 1n the specified dependent variable given a one-unit change 1n the 
Independent variable. For example, In Figure 4, the standardized path coefficient of .16, 
occurring above the arrow connecting aptitude with quality of performance, Indicates that an 
Increase In average quality of performance (dependent variable) of 16/100ths of a standard score 
unit would be expected to follow an Increase 1n average aptitude (Independent variable) of 1 
unit, when all other Independent variables are held constant. 

Inspection of Figure 4 Indicates that the student Input variables have substantial effects on 
assessed quality of student performance. The most powerful effects were generated by aptitude 
(.16), reading level (.16), and academic motivation (.14). These results suggest that 
Intellectual ability and motivation to achieve are prime determinants of training performance. 
However, sizable effects were also produced by age (.11) and educational level (.12), Indicating 
that maturity and educational exposure both have Independent causal effects on the quality of 
student performance. The weakest path coefficient produced by the various student Input 
variables was a coefficient of .07 associated with educational preparation. This suggests that 
although specific educational preparation m^y contribute to performance, In a well-designed 
training program 1t does not have a particularly strong Influence. 

A number of Interesting findings emerged among the course content variables. In defining 
course subject-matter difficulty, it appears that abstract knowledge requirements are the single 
best Indicator of this construct, although sizable coefficients were also obtained for reading 
difficulty, course diversity, and expected attrition rates, Indicating that all these variables 
must be considered for optimal definition of the construct. Similarly, 1n defining manpower 
requirements, both selective reenlistment bonus and yearly student flow yielded sizable 
coefficients. However, as Indicated by the substantially greater magnitude of Its standarlzed 
path coefficient, yearly student flow appears to be a more appropriate Index of manpower 
requirements. Manpower requirements and course subject-matter difficulty proved to be negatively 
related, as was expected from the correlational data. Moreover, course subject-matter difficulty 
was positively related to occupational difficulty, apparently because the difficulty of a 
training program reflects the difficulty of job tasks. Occupational difficulty, however, was 
positively related to manpower requirements, unlike course subject-matter difficulty. These 
results might be attributed to a greater civilian demand for individuals trained In more 
difficult occupations, and the limitations that difficulty of a training program places on 
student flow. 

In examining how course subject-matter difficulty, occupational difficulty, and manpower 
requirements determined lower-order course variables, a number of interesting findings emerged. 
It ms found that course length was determ1>°d primarily by occupational difficulty (.40) and 
subject-matter difficulty (.58), rather than by manpower requirements (.02). This finding could 
be expected in a system where Individuals must be trained to a constant standard of performance 
regardless of manpower requirements. On the other hand, 1t was found that Instructional day 
length tended to Increase with Increasing manpower requirements (.36) and decrease with 
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Figure 4 , Refined Model with Standardized Path Coefficients. 
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increasing course subject-matter difficulty (-.39). This set of path coefficients appears to 
support the hypothesis that an 8-hour Instructional da^y is used as one means of accommodating 
greater manpower needs, but that its use In difficult courses will be offset by the tendency of a 
longer Instructional day to overload students with Information. 

Student-faculty ratio was found to be negatively related to both subject-matter difficulty 
(-.38) and occupational difficulty (-.30), while It was virtually unrelated to manpower 
requirements (-.02). Thus, It appears that the number of students per Instructor Is primarily 
due to the difficulty of the job and course subject matter rather than manning needs. Instructor 
quality was found to be negatively related to subject-matter difficulty (-.10), occupational 
difficulty (-.30), and manpower requirements (-.47). Apparently, a high quality of Instruction 
Is less feasible when the class size is large and the course material Is difficult to teach. On 
the other hand, Instructor experience was found to be positively related to subject-matter 
difficulty (.42) and occupational difficulty (.19). These results confirmed the previous 
hypothesis that more difficult courses require and are typically assigned to more experienced 
Instructors. However, It was also found that Instructor experience tended to be limited by 
manpower requirements (-.17), Indicating that when need is high, it Is more difficult to obtain 
highly experienced Instructors. 

Feedback intensity was conceptually defined by the anount of feedback occurring in a course 
and the amount of practice provided. This construct was confirmed by the relationship of 
feedback Intensity with amount of feedback (.85) and practice (-.67). Feedback Intensity was not 
Influenced by course subject-matter difficulty (.04); however, it tended to be limited by both 
occupational difficulty (-.20) and manpower requirements (-.48). These negative effects might be 
attributed to the fact that It is more difficult for Instructors to provide frequent feedback 
concerning performance on difficult tasks and to the fact that a large number of students tends 
to restrict the amount of fe^back that can be provided. 

As defined In the refined model, instructional support was considered to be a function of the 
number of Instructional aids used and the amount of hands-on practice provided. This hypothesis 
was confirmed by the coefficient of .28 obtained for hands-on practice and the coefficient of .31 
obtained for the number of aids In use. Although the magnitude of these coefficients suggests 
that this latent variable was not as well defined ns the preceding latent variables, a consistent 
and interpretable pattern of path coefficients for this construct was generated by subject-matter 
difficulty (.44), occupational difficulty (.23), and manpower requirements (.10). As might be 
expected, Instructional support is most likely to occur when course training materials and job 
tasks are difficult. Furthermore, Instructional support tends to be employed more frequently 
w^en student flow Is sufficiently high to result In a payoff from its development. 

In addition to the above Interrelationships, it was found that the course content variables 
were capable of determining training outcomes. All three of the primal course content variables 
defined by subject-matter difficulty (-.08), occupational difficulty (-.13), and manpower 
requirements (-.10) had a direct causal Impact on the assessed quality of student performance. 
These effects indicate that difficult courses, difficult job tasks, and requirements to train 
large numbers of students will tend to restrict the quality of student performance. It was also 
found that SIA time was slightly Influenced by subject-matter difficulty (.16), by occupational 
difficulty (.06), and by manpower requirements (-.05). In fact, subject-matter difficulty was 
the prime determinant of SIA time, as was expected, since more difficult courses should cause 
progress check failure and therefore greater SIA time. Alternatively, manpower requirements had 
a negative effect on SIA time because large classes tend to restrict the amount of SIA that can 
be provided. 

When the effects of these three primal variables were controlled, it was found that certain 
lowe? -order course variables could also have an Impact on the assessed quality of student 
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performance and SIA time. In keeping with the interview reports, it was found that feedback 
p , intensity had the greatest impact on the quality of performance (.09) and SIA time (.08). The 

I former relationship may reflect the generally recognized facilitative effects of feedback on 

performance, whereas the latter relationship may Indicate that feedback can serve as an indicator 
of the need for SIA. As expected, instructor quality was found to have positive effects on the 
£ quality of performance (.05) and SIA time (.08). These results suggested that, good instruction 

[ enhances student performance and good instructors may be more likely to provide SIA time. 

^ Finally, as expected, course length had moderate positive impacts on both SIA time (.05) and the 

£ quality of student performance (.06). 

I The path coefficients obtained for instructional day length, student-faculty ratio, 

J: instructional support, and instructor experience were not especially powerful when the effects of 

\ course subject-matter difficulty, occupational difficulty, and manpower requirements were 

partialed out. However, both day length and student-faculty ratio had minor negative effects on 
i, assessed quality of performance and SIA time, supporting the information overload hypothesis and 

'v the notion that student performance had a slight tendency to decrease as class size increased. 

{*• Instructor experience, on the other hand, tended to have minor positive effects on both the 

|J quality of performance and SIA time, indicating that experienced instructors facilitate quality 

V performance and tend to provide more SIA. Finally, instructional support produced a minor 

positive effect for SIA time, perhaps because such support provides signals concerning the need 

for SIA. 

; As indicated in the preceding discussion, the assessed quality of stude/l performance had 

substantial effects on all relevant training outcomes, especially academic ccm.aeling (-.20) and 
nonacademic counseling (-.23). Academic counseling, in turn, had a positive effect on academic 
? attrition (.22) and retraining (.34)* while nonacademic counseling had a positive effect on 

; nonacademic attrition (.24). These results confirm the earlier hypothesis that counseling 

, moderates tha relationship between quality of performance and subsequent training outcomes such 

\ retraining and attrition. Further, a path coefficient between academic and nonacademic 

; counseling (.IS) supported the proposition that academic counseling tends to reveal nonacademic 

problems. A path coefficient between retraining time and academic attrition (.25) was also found 
J In this analysis. Thus, high retraining time appears to be an antecedent of attrition, as was 

I indicated by the interview data. 

<t 

The assessed queslty of performance exhibited a weak causal impact on SIA time (-.07). This 
suggests that poor performance might have some effect on SIA time* but that the difficulty of the 
? course material still appears to be the prime determinant. Finally, as expected, SIA time had a 

direct Impact (.18) on academic counseling. This result confirms the expectation that repeated 
SIA sessions tend to serve as a signal for academic problems. 

Overall, the foregoing discussion of the standardized path coefficients resulting from the 
analysis indicates that the refined model yields a highly meaningful and interpretable set of 
interrelationships. This provides compelling evidence for the validity of the model when coupled 
I with the observed residual, goodness-of-fit test, and Rs. The refined model, with regression 

{. coefficients and total effects, is presented without further discussion in Appendices J and K, 

respectively. 

i 

One last topic that must be addressed concerns one of the basic assumptions of path 
I modeling. The Independence of errors associated with variable measurement is one of the more 

I Important assumptions associated with this analytical technique. Evaluation of correlations 

between the error terms for the course variables, as well as correlation? between the error terms 
of the course variables and training outcome variables, served as a basis of determining the 
1 tenabillty of this assurrption. These correlation coefficients are provided in Appendices L and 

M» respectively. Inspection of these data indicates that the error terms of the course 




parameters and the outcome variables were not strongly related, and that the error terms among 
the course parameters themselves were effectively Independent* Correlations between the error 
terms for the training outcome variables which are presented <n Appendix N Indicated a pattern 
which conforms to the formal path model* For instance, errors In the measurement of the assessed 
quality of performance were slightly related to errors associated with other training outcomes* 
However, this result was expected because assessed quality of performance is the basis for 
decisions associated with other training outcomes such as retraining and attrition* Similarly, 
measurement errors for academic counseling, which plays a pivotal role In these training outcome 
decisions, were correlated with all other variables* Thus, the pattern of relationships among 
error terms Is consistent with the overall model. 



Cross«Va11dat1on 

The preceding discussion has demonstrated that the refined model provides a meaningful and 
interpretable description of the relationships among student input, course content, and training 
outcome variables and that it accounts for much of the variance in training outcomes* Although 
these results clearly suggest the predictive utility of the model, the questions that arise at 
this point are whether accurate predictions will result when courses not employed iu model 
development are examined, and most Importantly, whether the refined model is sufficiently 
representative to be effectively employed in making predictions for a diverse set of courses* 

To answer these questions, a cross-validation analysis compatible with the path-analytic 
procedures was carried out* This analysis entailed obtaining standard scores on the student 
Input and course content variables for each of nine courses held for cross-validation and 
entering values for the student Input and course content variables for each course Into the 
relevant regression equations to obtain standard score predictions for the seven training 
outcomes* These standard score predictions were then converted to raw score predictions* Once 
this information was obtained, It was directly compared to the raw scores and standard scores 
actually observed for the cross-validation courses for each training outcome* 

It should be noted that when generating training outcome predictions for each course, the 
nature of the model dictated the application of a certain strategy* Distal training outcomes are 
not predicted directly from knowledge of student Input or course content variables* The refined 
.nodel holds that distal training outcomes, such as attrition and retraining, are dependent on the 
intervening counseling process and assessed quality of performance* Thus, predictions of the 
assessed quality of performance and SIA time, which Issued directly from knowledge of student 
Input and course content variables, were used In deriving , 'edictlons of academic and nonacademic 
counseling rates* In turn, the predicted academic and nonacademic counseling rates were used in 
conjunction with the appropriate regression weights to generate predictions of academic 
attrition, nonacademic attrition, and retraining time* Although this procedure was In keeping 
with the design of the model and Its intended application, it should be borne In mind that due to 
the possibility of cumulative prediction error, It constitutes a conservative test of the 
predictive accuracy of the model with respect to the distal training outcomes of academic 
counseling, nonacademic counseling, retraining time, academic attrition, and nonacademic 
attrition* 

Table 6 displays observed and predicted scores In both raw score and standard score form for 
all training outcomes for each of the nine cross-validation courses* As may be seen, predictions 
of assessed quality of performance, which was expressed as an average final course grade, 
approximated the observed values* Across the nine courses, there was an average absolute 
difference between predicted and observed raw scores of 2*34 points* It is noteworthy that the 
model predicted average final school grade about equally well for all nine courses* In fact, the 
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Table 6 . Predicted and Observed Training Outcomes 



Assessed quality of performance SI A time Academic counseling 



Course 


Raw scores 


Standard 
scores 


Raw scores 


Standard 
scores 


Raw scores 


Standard 
scores 




JOBS 


PREP 


OBS 


PREP 


OBS 


PREP 


OBS PREP 


OBS 


PREP 


OBS PREP 


Aircraft Electrical Sys. Spec. (42330) 


90.0 


86.5 


.64 


.16 


3.8 


10.8 


-.13 .32 


2.3 


1.5 


.26 .04 


Jet Engine Mechanic (42632) 


85.0 


81.5 


.00 


-.45 


4.6 


10.7 


-.08 .32 


.7 


1.4 


-.20 .01 


Financial Management Spec. (67231) 


86.4 


88.2 


.19 


.41 


10.3 


5.9 


.28 .00 


1.3 


1.2 


-.03 -.07 


Medical Laboratory Spec. (92430) 


84.9 


89.5 


.00 


.59 


26.4 


10.1 


1.21 .28 


2.5 


1.2 


.32 -.05 


Armament Systems Spec. (46230) 


85.9 


85.3 


.11 


.05 


1.5 


8.1 


-.28 .15 


.8 


1.5 


-.15 .03 


Ground Radio Operator (29333) 


86.6 


86.6 


.20 


. .20 


12.5 


6.0 


.43 .01 


2.8 


1.3 


.24 .29 


Aircraft Warning Radar Spec. (30332) 


89.5 


88.3 


.58 


.43 


13.1 


10.9 


.32 .48 


2.1 


1.5 


.20 .20 


Navigation Systems Spec. (32831) 


90.5 


88.1 


.74 


.39 


3.3 


10.6 


-.17 .33 


1.4 


1.4 


.00 .00 


Computer Programming Spec. (51131) 


87.4 


90.4 


.37 


.71 


5.8 


10.0 


-.01 .27 


.6 


1.1 


-.23 -.07 


Average Absolute Difference 8 


2.34 


.331 


6.83 


.421 


.67 


.182 



Nonacadewlc counseling Retraining time Academic attrition Nonacademlc attrition 





Raw 

scores 


Standard 
scores 


Raw 

scores 


Standard 
scores 


Raw 

scores 


Standard 
scores 


Raw 

scores 


Standard 
scores 




OBS PRED 


OBS 


PREP 


OBS 


PREP 


OBS 


PREP 


OBS , 


PREP 


OBS 


PRED 


OBS 


PRED 


OBS 


PRED 


Aircraft Electrical Sys. Spec. (42330) 


.10 


.13 


-.06 


-.03 


8.8 


9.3 


.00 


.01 


.02 ' 


.02 


.00 


.013 


.000 


.003 


-.07 


.021 


Jet Engine Mechanic (42632) 


.16 


.26 


-.01 


.0B 


16.5 


10.5 


.18 


.04 


.03 


.02 


.06 


.011 


.010 


.006 


.10 


.069 


Financial Management Spec. (67231) 


.05 


.05 


-.09 


-.09 


25.0 


7.7 


.37 


-.02 


.06 


.02 


.26 


-.OK 


.000 


.002 


-.07 


-.023 


Medical Laboratory Spec. (92430) 


.11 


.00 


-.05 


-.12 


53.7 


8.0 


1.00 


-.02 


.11 


.01 


.60 


-.017 


.000 


.002 


-.07 


-.032 


Armament Systems Spec. (46230) 


.07 


.14 


-.08 


-.01 


14.9 


9.3 


.14 


.01 


.05 


.02 


.20 


.010 


.000 


.003 


-.07 


-.001 


Ground Radio Operator (29333) 


.03 


.10 


-.10 


-.05 


28.0 


8.3 


.44 


-.01 


.06 


.02 


.26 


-.011 


.000 


.003 


-.07 


-.012 


Aircraft Warning Radar Spec. (30332) 


06 


.06 


-.08 


-.08 


13.9 


9.1 


.11 


.01 


.04 


.02 


.13 


.015 


.000 


.002 


-.07 


-.022 


Navigation Systems Spec. (32831) 




.08 


-.09 


-.07 


14.0 


9.0 


.12 


.01 


.01 


.02 


-.06 


.003 


.000 


.002 


-.07 


-.014 


Computer Programming Spec. (51131) 


.02 


.00 


-.10 


-.14 


9.2 


7.7 


.02 


-.02 


.00 


.02 


-.12 


-.024 


.000 


.001 


-.07 


-.037 


Average Absolute 01fference a 


.64 


.05 


11. 


62 


.27 


.028 


.1 


72 


.002 


.031 



Observed versus Predicted Scores. 
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largest difference observed between predicted and observed raw scores on this measure was only 
4.6 points. 

The model's accuracy In predicting assessed quality of performance was not matched w';h 
regard to SIA time. The overall difference between predicted and observed raw scores for 'die 
nine cross-validation courses was 6.83 hours. These results were expected, given that SIA time 
produced the smallest R observed In the present Investigation. 

Table 6 also presents the results obtained In the cross-validation analysis for academic 
counseling and nonacademlc counseling. For the nine courses, the average difference between 
observed and predicted raw scores was .67 for academic counseling, and .64 for tt.e nonacademlc 
counseling. The largest difference In observed and predicted raw scores was .11 for nonacademlc 
counseling and 1.5 for academic counseling. Given the relative Infrequency of nonacademlc 
counseling, It was surprising that the model provided more accurate predictions for It than for 
academic counseling. However, this result might be attributed to the fact that within the model, 
cumulative errors carried over from SIA time would affect predictions of academic counseling more 
than they would predictions of nonacademlc counseling. 

With respect to retraining time, the average difference obtained between predicted and 
observed raw scores was 11.6. Predictive accuracy held across courses, with the exception of 
those that had unusally high retraining times. The results obtained for academic attrition 
Indicated that the average difference between predicted and ooserved attrition rates was .03. 

As was noted earlier, the nonacademlc attrition rate was extremely low. In fact, for the 
nine cross-validation courses, only one student was eliminated for nonacademlc reasons. This 
nonacademlc student elimination occurred In the Jet Engine Mechanic course. Surprisingly, the 
'highest predicted nonacademlc attrition rate was for this course, and In fact, It was the only 
predicted value which was not extremely close to zero. 

Clearly, the predicted outcomes were reasonably consistent with the actual values. However, 
the cross-validation study does not fully demonstrate general Izablllty of results across courses, 
time, and both courses and time. Therefore, we think It best to view the present model as a 
reasonably good first approximation to the complex relationships Involved In Initial-skills 
training. As additional data become available and as personnel and training managers gain 
experience with the working of the system, It would be appropriate to revisit empirically the 
accuracy of the model 1 s predictions for the same courses at different points In time, for 
different courses, and for different courses at different points In tiMe. 



IV. DISCUSSION 

Findings 

The principal concern giving rise to this work was the need for some technique that would 
allow estimation of the effects of aptitude requirement adjustments on technical training 
outcomes prior to actually making such adjustments. Because the model described above has been 
shown to be quite stable and to have substantial predictive power, It appears to provide a viable 
basis for drawing general conclusions concerning the relationship between aptitude and training 
outcomes. 

The most straightforward conclusion indicated by the content of the refined model is that 
aptitude is clearly a prime determinant of training outcomes. This was attested to by the fact 
that aptitude yielded one of the most powerful path coefficients against assessed quality of 
student performance. Nevertheless, this observation should not be taken as Indicating that 
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aptitude Is the only determinant of training outcomes* The refined model clearly indicates that 
a variety of other student Input variables, sue** as reading level, academic motivation, 
educational level, and educational preparation ar* *ii associated with sizable path coefficients, 
Indicating that they too have a significant impact on assessed quality of student performance. 
This Is not an especially surprising finding since a wealth of literature exists Indicating that 
motivation to perform and prior educational achievements are often critical determinants of 
training performance. Moreover, the magnitude of the coefficients associated with academic 
motivation, educational level, and educational preparation seem to indicate that in constructing 
a manpower allocation system of maximum effectiveness, these Indices of student quality should be 
taken into account, Further, It appears that there might be substantial value in considering 
seme Index cf maturity, such as age, given the sizable path coefficient produced by this 
variable. Although an attempt was made to examine Interest, the available measures of this 
variable made It impossible to effectively assess the Impact of interest on the assessed quality 
of student performance. Thus, this Issue must remain open until more effective and appropriate 
Interest measures become available. 

Tn addition to recognizing that aptitude Is only one of a number of variables that influence 
the quality of r*uder.t performance, it must also be recognized that aptitude and its effects are 
in part Imbedded in relationships with a number of other variables. Thus, when considering the 
potential Impact of aptitude requirement adjustments, one must view these effects as being 
complex, multifaceted effects that are exerted through a variety of closely related variables. 

One of the more interesting and significant findings generated by the model is the manner in 
which aptitude, or for t.hat matter ar*y student Input variable, Influences training outcomes. 
Aptitude and other student inputs have a strong direct Impact on the assessed quality of student 
performance. However* the model Indicates that student inputs have no direct effect on dictal 
training outcomes such as academic attrition, nonacademic attrition, and retraining. Rather, 
student Inputs affect the assessed quality of student performance, which in turn affects either 
academic counseling or nonacademic counseling. Retraining time, academic attrition, and 
nonacademic attrition represent potential negative outcomes of these counseling sessions. 

This set of findings appears to conform to current operational procedures as they occur in 
the initial-skills training programs administered by ATC. For example, the fundamental 
importance of academic and nonacademic counseling in affecting distal training outcomes is an 
Important finding which has a number of implications. First, it indicates that whether an 
individual is eliminated from training or retrained is ultimately in the hands of course 
Instructors. Second, given the fundamental importance of instructors* counseling decisions in 
determining retraining and attrition, it appears that more attention should be given to this 
process In future research efforts, and that both academic and nonacademic counseling should be 
considered major training outcomes. Third, given the apparent importance of this process in 
mediating the relationship between performance and distal training outcomes, consideration* 
underlying trainers* counseling decisions should receive attention not only in research efforts, 
but also with regard to policy formulation. 

Two other points concerning the relationships observed among training outcome variables 
should be elaborated. First, although student Inputs affect assessed quality of performance, 
which in turn Is the fundamental determinant of counseling and attrition, the model indicates 
that a clear-cut distinction Is made between purely academic performance problems and academic 
performance problems resulting from nonacademic causes such as a lack of motivation. Thus, a 
careful distinction should be made between academic and nonacademic outcomes despite their common 
cause. It should be noted that the present findings suggest that academic problems are far more 
common than nonacademic problems, and that academic counseling at times reveal nonacademic 
problems as indicated by the causal path from academic to nonacademic counseling* 
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The second point Is that while SIA time represents a significant outcome variable, 1t Is 
somewhat unique. SIA time was not found to be highly related to either student Inputs or the 
assessed quality of student performance. Rather, SIA time was primarily driven by the difficulty 
of course subject matter and occupational difficulty. Thus, It appears that SIA time Is truly 
employed as a remediation vehicle. However, the model also Indicates that a repeated need for 
remediation may affect academic counseling and, in turn, academic attrition and retraining 
decisions. This suggests that though SIA Is a somewhat unique variable, it is of some Importance 
to conceptualization and prediction of training outcomes. 

As was Implied in the discussion of SIA time, the various course content variables also 
produced some Interesting relationships. Broadly speaking, the refined model hypothesizes that 
training Is driven by three primal variables: course subject-matter difficulty, occupational 
difficulty, and manpower requirements. That Is, the model holds that courses are designed on the 
basis of the difficulty of the job, the difficulty of course training materials, and the number 
of students to be trained. This conceptualization of the training process ti's quite well with 
the Information obtained through the Interviews, where it was noted that the job demands, yearly 
student flow, and training materials were the major factors considered In course design. 

Of these three primal variables, only occupational difficulty was measured directly. 
Manpower requirements and subject-matter difficulty represent latent variables defined In terms 
of directly measured variables. Subject-matter difficulty was defined by the diversity of course 
inaterlal, the reading difficulty of training materials, expected attrition rates, and abstract 
knowledge requirements. Of these four variables, abstract knowledge requirements exhibited the 
largest loading for subject-matter difficulty, suggesting that abstract knowledge requirements 
are important fo~ training design and training outcomes. Manpower requirements were defined by 
selective reenlistment bonus and yearly student flow. As expected, the path coefficients 
indicated that yearly student flow was the strongest index of manpower requirements. 

Although It may be difficult to conceive of these three relatively straightforward variables 
driving course design^ this hypothesis was strongly supported by the results obtained in the 
present effort. For instance, these three variables yielded Rs for the prediction of course 
length, day length, student-faculty ratio, instructor quality, Instructor experience, 
instructional support, and feedback intensity in excess of .54 and averaging near .60. Thus, 
these three primal variables provide an exc^lent definition of lower-order course variables. 
Moreover, the nature of the paths between these primal variables and the lower-order course 
variables was readily interpretable, thus supporting the validity of this conceptualization. 

Two latent variables were among the lower-order course variables held to be driven by the 
three primal variables. The first was instructional support, which was defined by a combination 
of the frequency of use of instructional aids and the number cf hours of hands-on instruction. 
This variable reflects the frequency of use and the amount of course time devoted to job task 
simulations. The second latent variable, feedback intensity, was defined by the amount of 
feedback and practice given a student. The relation of practice to feedback Intensity may, at 
first glance, appear ambiguous. However, the relation Is based on the fact that feedback in 
basic resident technical training is required by unit of instruction. Thus, less time per unit 
equates to more rapid feedback. 

Within the model, these seven course content variables were hypothesized to be capable of 
Influencing assessed quality of performance and SIA time independently of the three primal course 
variables. With respect to the magnitude of these effects, course length, Instructor quality, 
and feedback Intensity had the greatest Impact on both SIA time and the quality of student 
performance. This suggests that more rapid feedback, a better overall quality of Instruction, 
and more Instructional time enhance the quality of performance* On the other hand, the patns 
obtained for instructor experience, day length, student-faculty ratio, and instructional support 
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were relatively weak. This observation should not be taken as Indicating that these latter 
variables are irrelevant to student performance. Rather, the results Indicate that these 
variables have relatively less Impact on training outcomes when the Influence of the three primal 
course content variables 1s partlaled out. This suggests that if any given course content 
variable 1s out of line with the program's status on course subject-matter difficulty, 
occupational difficulty, or manpower requirements, it might have a marked Influence on the 
quality of student performance. 

To this point, little attention has been given to the Impact of the three primal course 
content variables on training outcomes. As the standardized path coefficients Indicate, course 
subject-matter difficulty, occupational difficulty, and manpower requirements all have 
substantial effects on both assessed quality of student performance and SIA time. In fact, 
course subject-matter difficulty had the greatest effect on SIA time. It should also be pointed 
out that the primal course content variables generally had a greater impact on assessed quality 
of student performance than any other course content variable. Thus, it appears that the three 
primal course variables not only drive course design, but also constitute those course content 
variables most likely to Influence student training performance. 

One question that might arise at this point concerns the relative power of the course content 
and student input variables in determining assessed quality of performance. Overall, evaluation 
of the standardized path coefficients tends to indicate that the course content and student input 
variables make roughly comparable contributions to the prediction of assessed quality of student 
performance, with the student input variables showing a slight advantage. This 1s not an unusual 
finding, since previous research (Tyler, 1964) has Indicated that what the Individual brings to a 
course has a more Important influence on educational outcomes than characteristics of the 
educational system. On the other hand, 1t should be recognized that course content variables did 
have a substantial impact on the assessed quality of performance. Taken as a whole, the 
foregoing considerations Indicate that, although student inputs may be the most effective 
predictors of performance, both student Inputs and course content variables should be considered 
1n order to attain optimal performance prediction. 

Before concluding this discussion, one additional set of comments seems in order. At first 
glance, t*e path coefficients obtained for many of the variables examined in the present 
investigation may seem low and might be viewed as Indicating a lack of effective prediction. 
However, the size of the Rs generated by the model for each of the outcome variables was quite 
large, ranging from .50 for nonacademlc attrition to .83 for academic attrition. 

If one grants that the model provides a stable and accurate description of the training 
process, then the magnitude of the Individual path coefficients can be viewed in a different 
lit^t. Rather than diminishing the appropriateness of the model, they suggest that no single 
variable or limited set of variables 1s adequate for predicting training outcomes. This 
observation, 1n turn, confirms the initial premise of the current investigation that resident 
technical training is a highly complex process which can be adequately described only by 
considering a variety of student input, course content, and training outcome variables. Thus, 
the model developed in the present effort Indicates that fully adequate predictions of training 
outcomes can be obtained only when training is conceived of as being a complex, multlfaceted 
process involving, a variety of student Inputs and course content variables. AT hough this is not 
an especially unusual conclusion, it suggests that optimal description and prediction of the 
training process can never be attained without employing multivariate techniques capable of 
taking into account these complex interrelationships. Perhaps much of the value of the present 
effort lies 1n its recognition of this complexity, and its use of a heuristically appropr'ite 
modeling methodology to demonstrate the complexity of basic resident technical training. 
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Applications 



The discussion has focused primarily on the empirical model of resident training developed 1n 
the present Investigation, along with some of the conclusions that flow from the content of the 
model. However,, since the model was developed for the purpose of making predictions concerning 
the Impact of aptitude requirement adjustments on technical training outcomes, 1t Is necessary to 
conclude the discussion by considering the potential applications of the model. 

61ven that the model provided both an accurate description of the 1n1t1al-sk1lls training 
process, as well as accurate predictions of training outcomes, a technical constraint that might 
limit application of the model must be examined. To employ the model on a routine basis for 
predicting training outcomes, 1t will be necessary to specify the status of the course on the 
various student Input and course content variables. As a result, routine application of the 
raod.el 1n the prediction of training outcomes requires that measures of student Input and course 
content variables are readily available. Anticipation of this constraint resulted 1n the use of 
measures of student Input and course content variables that are easily obtained from sources such 
as the PACE data file and standard training documents such as the POI, training plan, and other 
course materials. The only exception 1s the ratings of course abstract knowledge requirements 
made by the ARRO staff. However, given the simplicity of the measurement procedure for this 
variable, there would be little difficulty 1n using 1t to make similar ratings when the need 
arises. 

The foregoing discussion addressed the major technical issues associated with application of 
the model. Therefore, 1t would now seem appropriate to turn to some of the specific applications 
of the model. Assuming the availability of requisite information describing student input and 
course content variables, the model could be used 1n a straightforward fashion to predict the 
•effects of aptitude requirement adjustments on technical training outcomes. For a specified 
course, the average aptitude of Individuals assigned to the associated job specialty on the basis 
of a given aptitude requirement minimum would be obtained, along with average values fo<* other 
student Inputs and values which describe the course's status on course content variables. These 
values would then be converted to standard scores and used with a series of mathematical 
equations to predict average values on the training outcome variables. These equations would 
produce predictions of training outcomes in standard score form. To facilitate interpretation, 
it would be desirable to convert the standard score predictions back to raw-score form. The 
whole student input model could be manipulated by adjusting the means of certain student Inputs 
to obtain training outcomes at desired levels. 

In addition to the applications of estimating student inputs required for entry into a 
training program, the model has a number of other uses. For instance, if a simulated change 1n 
aptitude mlnimums results 1n a predicted decrease in the assessed quality of student performance, 
either the primal course content variables or the lower-order course variables might be 
manipulated to compensate for the predicted decrease in performance. Hence, the model may be 
used not only to predict the effect that changing student Inputs has on training outcomes, but 
also to suggest specific training strategies for compensating for negative effects of these 
changes. 

In addition to compensating for changes 1n student inputs, the model can provide a framework 
for course design by evaluating changes 1n course subject-matter difficulty, occupational 
difficulty, or manpower requirements. Values for these primal course variables might then be 
used to predict expected values for lower-order course variables, such as course length or 
student-faculty ratio. Predicted values for these latter course variables as they follow from 
changes 1n job difficulty, course subject-matter difficulty, or manpower requirements might 
provide a basis for decisions concerning course design. Moreover, because the model specifies 
the relationships of the primal course content variables and the lower-order course variables 
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with the various training outome variables, the potential impact of proposed changes in course 
content on training performance can be predicted. If changes in course content result In 
predictions of decreased training performance, an attempt might be made to manipulate other 
course content or student input variables so as to avoid such a decrease. 

Obviously, the model has many potential applications, Including prediction of the effect* of 
changes in aptitude requirement mlnimums and modifications in course design on training 
outcomes. However, an issue that Is likely to arise in such applications is related to the 
Riultlvariate nature of training outcomes. In the refined model, there are five training 
outcomes, specified as academic attrition, nonacademic attrition, retraining time, academic 
counseling, and nonacademic counseling, all flowing from the assessed quality of student 
performance and SIA time. Thus, In evaluating the effects of any simulated change in student 
input or course content variables, a variety of potential outcomes must be considered. Because 
multiple outcomes must be considered, It would be desirable to have some means of combining these 
outcomes into a single decision-making index. One way of accomplishing this Is to estimate the 
costs associated with training outcomes. The resulting cost estimates would then provide a 
common framework for evaluating changes In student input or course content. Because the ATC 
Comptroller maintains training cost data for each course, It may be possible to employ these 
training cost data In the implementation of the model. Therefore, changes in training outcomes 
could be linked to training costs, and thereby serve as a basis of evaluating the overall cost 
impact of student Input or course content adjustments. 

This has been a fairly general discussion of potential applications of the model formulated 
in the present effort. This discussion has underscored the point that the model will not only 
serve the intended purpose of predicting the effect Jf aptitude requirement adjustments on 
technical training outcomes, but can also serve a variety of other purposes as well. Perhaps 
more important than these relatively pragmatic applications, the study may have made one other 
major contribution. By defining the variables relevant to technical training outcomes and 
establishing their Interrelationships, the model provides a basis for facilitating understanding 
of the initial-skills training process as a whole. 



The present effort was undertaken as part of a larger Air Force effort concerned with the 
enhancement of manpower allocation. More specifically, it was concerned with finding some 
technique for assessing the Impact of changes in aptitude requirement minimums on technical 
training outcomes. To accomplish this, it was decided that an empirical model should be 
developed by which student input and course content variables could be related to training 
outcomes. 

A series of interviews were conducted with technical training instructors and administrators 
to identify the student input and course content variables most likely to influence performance 
in basic resident technical training. Readily available measures of these variables were then 
specified. Subsequently, data describing the status of some 5,000 trainees in 39 courses were 
obtained, along with data for some 1,000 additional trainees In nine different courses. Path 
modeling procedures were used to develop an empirical model of the technical training process. 

The resulting model provided an effective description of the relationships among student 
input, course content, and training outcome variables. The observed residual (variance 
unaccounted for) was only .19, and the model yielded an R of .75 against the assessed quality of 
student performance. 



V. CONCLUSIONS 
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A number of Interesting facets of the technical training process were Identified during model 
development. For instance, although aptitude was one of the best predictors of the quality of 
student performance, it Interacted with a variety of other student input and course content 
variables in determining training outcor*s. Further, it appeared that course content variables 
tended to be defined by course subject-matter difficulty, occupational difficulty, and manpower 
requirements. Another Important finding was that finally, the model indicated that counseling 
mediated the relationship between training performance and distal training outcomes such as 
retraining time and student elimination. 

Results of the analyses Indicated that the mulel could be used both to predict the effects of 
aptitude requirement adjustments and to enhance training course design. Moreover, It appeared 
that only a multivariate modeling approach, such as that employed In the present investigation, 
would allow effective predictions of Air Force Initial-skills training outcomes. 
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APPENDIX A i INTERVIEW PROTOCOL 



Sample of Questions Asked Technical Training 
Center Staff Members 



Course Content 

What determines course content? 

How 1s the length of a course determined? 

How much Instructional System Development (ISD) effort Is done? 

In the past couple of years, have there been ar\y changes In the courses you are connected with 
(e.g., changes In content of courses; number of class hours per day; length of course; entry 
aptitude requirements)? 

How are these changes accomplished? 

What has been the effect of these changes on student performance, anount of recycling, attrition, 
quality of output, etc.? 

If you encountered changes 1n the general aptitude level of your students, what would you do to 
preserve the quality of your product? 

What changes would you like to see made in your courses and why do you think these changes would 
be beneficial? 

How do student aptitudes affect performance in your courses? Is it very noticeable? 

What changes were made in other aspects of the course to compensate for the changes 1n aptitude? 

Have there been ar\y changes in aptitude requirements to your courses? Where Is a record of these 
changes maintain^ 4? 

What differences have you noticed between non-prior-service students and those who are being 
retrained? 

If no changes in aptitude requirements have been made, do you feel that changes in aptitude 
requirements should be made? Why? 

Besides aptitude, what personal characteristics of students are most influential on their 
performance In training? 

How dc students' educational experiences affect their performance (e.g,, the kind of courses 
taken)? 

Which do you prefer: group lock-step or group-paced courses? Why? 
How should the difficulty of a course be measured? 

Can the difficulty of a block of instruction be estimated from Occupational Survey Report (OSR) 
task difficulty ratings? 
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What aspects of a course have the most effect on Its difficulty? 
How do you get feedback about your courses? 
How Is this information used? 

Is It possible to use Career Development Course (CDC) completion as a measure of graduates 1 
quality? 

Student Assessment 

How Is student performance measured? 

What are the differences between progress checks, performance tests, sampling written tests, and 
comprehensive written tests? 

How are progress checks and performance tests conducted? 
What determines which kind of measurement is to be used? 
How and when are these measurements made? 
What records are required to be kept on student performance? 

Are there any important aspects of student performance that are not kept on the ATC Form 156? 
How useful Is the ATC Form 667 in recording progress checks? 

How much consistency Is there across Instructors In grading progress checks or performance tests? 

Have you ever retested students on block tests to measure reliability of tests? 

What factors influence the grading of progress checks or performance tests? 

How are these assessments combined into an overall assessment of student performance? 

How Is a student's final school grade (FSG) computed? 

What Is the usual distribution of FSGs? 

Are any analyses made of FSG distributions? 

What feedback is provided as a result of these analyses? 

How are distinguished or honor graduates chosen? 

What Is the difference between the top students and the poorest students? 

Where and how do these differences show up? 

What are the main causes of poor student performance? 

How well Informed are students about the nature of the work they will be doing In the job 
specialty for which they are training? Does this affect their school performance? 
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Are those In the Guaranteed Training Enlistment Program (GTEP) better Informed or motivated? 

Do you have any courses where the nature of conditions of work tends to create stress among the 
students? 

What Inducements can lead to better student performance? 

How do you determine when a student needs Special Individual Assistance (SIA) ? 

What determines how much SIA Is given? 

Should SIA be considered as additional training time or Is It a different type of training? 

Where Is the information concerning SIA recorded and how accurate 1s It? 

Who uses the SIA Information and how 1s It used? 

What determines when a student Is recycled or washed back? 

Where 1s Information concerning trainee wash-back recorded and how 1s It used? 

What determines when a student should be failed? 

Ooes the policy toward attrition vary over the months or years? 

What are the reasons for attrition and how do you determine which reasons to assign to a 
particular case? 

What factors have the most effect on the various categories of attrition? 
Course Quality 

How is th? quality of a course measured? 

Where may information concerning course quality be obtained? 

What regulations or directives govern the compiling and maintenance of course quality records? 

Which are better Indicators of course quality? 

How and by whom Is information concerning course quality used? 

How far back do records concerning course quality go? 

How are student attrition figures computed? 

What kind of feedback concerning course quality do you get from the field? 

How useful are student critique forms? 

Instructor 

What effect does an instructor have on how much a student learns? 
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In what Mays do Instructors Influence student performance? 
On the average, how many Instructors teach In a single course? 
How Is the required number of Instructors determined? 

Are there times when there aren't as many instructors available as are required? 
ATC Form 281 Is used to evaluate instrurtnr* nf fh« 

the most instant? Which the least 5j5SS ' * f ° m ' Which are 

How should we measure Instructor quality? 

How does an instructor's field experience or lack thereof affect his/her performance? 
What are the characteristics of good and poor Instructors? 
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APPENDIX B : STUDENT MEASUREMENT GUIDELINES 
(ATCR 52-3) 
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DEPARTMENT OF THE AIR FORCE ATCR 520 

Headquarters Air Training Command 

Randolph Ai' Fore: Base, Texas 78150 25 July 1982 

Technical Training 

STUDENT MEASUREMENT 

This regulation establishes policy, assigns responsibilities, prescrihes procedures, and provides guidance for student 
measurement in technical and military training programs consistent with AFM 50-2. It applies to all technical training 
wings, 3785FJ.DTG, School of Health Care Sciences, 3480TCHTG. and the Basic Military Training School. 



1. Terms Explained: 

a. Appraisals. A group of questions and (or) projects 
used to check the day-to-day learning process. Appraisals 
are an informal, optional measurement and are not a part 
of the recorded measurement program. 

h. Comprehensive Written Test. Written test items 
used to fully measure student achievement of a knowledge 
oriented objective(s) not included on progress checks. 

c. Compromise of Test Materials. Any act or 
occurrence wherehy students gain unauthorized knowledge 
of contents of test materials. 

d. Criterion Checklist. A reference or listing on A'lC 
Forms 667 or 667A, Criterion Checklist, of course 
objectives that are designed to be assessed b> progress 
checks, in courses without a POI, objectives are extracted 
from lesson plans and listed on the criterion checklist. 

e. Criterion Referenced Testing (CRT), lest to 
determine if the behavior specified in the objective has heen 
acquired. May involve multiple-choice, completion, true or 
false, matching, essay items, short answer or actual 
performance of a task. 

f. End-of-Course Test. Written test items 
administered near the end of a eoutse to cheek the retention 
of the knowledge component of selected objectives an 
optional quality control check. 

g. Master Copy of Written Test. A copy of each 
written test annotated to ( I ) identify the correct answers, (2) 
show POI objective reference which each item supports, 
and (3) list the approved corrections and changes made in 
the test, it must also contain a statement showing approval 
for continued use when the test date is earlier than the POI 
date, or the course chart date for courses that have no POI. 

h. Measurement Plan. A plan designating the method 
used to measure student achievement ol the course 
objectives and indicating the correlation between test items. 
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and POI ohjectives and the correlation hewecn PO 
objectives and training standard items. 

i. Objective. A precise statement of the student* 
behavior to be exhibited, the minimum standard o 
performance or proficiency expected and the condition 
under which the behavior is to he exhihitcd (AFM 50-? 
attachment I). 

(1) Knowledge Oriented Objective. An objectiv. 
originating on the basis of a training standard elements 
requiring task or suhject knowledge. Conditions am 
standards may be implied if they are clearly obvious. 

(2) Performance Oriented Objective. An objectiv 
originated on the hasis of a training standard elements 
task performance. Standards may be implied if clear!; 
obvious. 

j. Performance Exercise. The ..ans of providing th 
students practice sessions throughout the learning period 

k. Performance Test, An instrument administered in 
test mode and environment after the teaching-learnin 
aciivit) to verify student attainment of pcrformanc 
oriented ohjectives. 

I. Performance Recheck. A rceheck of studen 
performance on a specific objective, normallv administers 
near the end of the course, it is mandatory when a specie 
requirement has heen identified in the training standard o 
where a separate agency certification is required. It i 
optional when objectives require added quality control. 

m. Progress Check. The assessment of student ac 
complishment o* knowledge or performance objective 
during the teaching-learning activity. Results ar 
documented on ATC Form 667 or 667A. 

n. Progress Checklist. (ATC Form 98 or locall 
approved checklist) The breakdown of an objective int 
elements or steps necessary to determine whether eac 
student satisfactorily accomplishes an objective during 
progress check or performance test. 

o. Sampling Written Test, instrument used to sampi 
the knowledge components of each course objectiv 
assessed during a progress check. 

p. Test Item Pool. A file of written test items which ea 
be used to construct or revise a test. Items ma)** b 
handwritten, be a copy of a test replaced by a nevvc 
revision, or a copy of a test containing annotated propose* 
changes (working copy). Maintenance of a test item.pool i 
optional. • 

q. Test Materials. Tests or pages from a test, to 
booklets, scoring keys, test items in pools, complete 
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answer sheets, test data forms, automatic data processing 
cards as we)) as calculations, diagiams. formulas, i.e., used 
by a student while being tested. 

r. Training Deficiency. A condition that prevents 
accomplishment of training requircmcnt(s) and which 
cannot be corrected before the affected .students graduate. 

2. Student Measurement Concept. The ATC technical 
training student measurement program is predicated on 
valid, joh relevant % and properly constructed objectives. 
Since the ohjectives specify precisely what behavior is to be 
exhibited, the condition under which the behavior will he 
accomplished, and state or imply the minimum standard of 
accept a hie aehavior, they are the basis for measurement of 
student achievement. Finch criterion test item is hascd sole!) 
on the requirements specified in the ohjective which it is to 
measuie. This program normally includes instructor 
assessment of tach student's accomplishment of each 
objective during the teaching-learning environment 
through the use of progress checks. Progress checks are the 
most practical means of verifying accomplishment of each 
course objective. However, when it is more appropriate to 
delay the measurement of certain objccti\cs and satisfy 
them through the use of comprehensive written and (or) 
pill oi i nance tests, stiuctuie the measurement de\ice in a 
manner which determines students' achievement of each 
object i\e tc:.:cU. 1 he object i\es assessed by a progress check 
during the teaching-learning activity are sampled on unit, 
block, and (or) end-of-coursc written tests to aid in 
retention and act as quality control devices. Additional!), it 
Is often appropriate to rcchcck certain performance 
oriented ohjectives with performance rechecks. Normative 
grading systems (grading on a curve) aie not used. 



3. Objectives o! Student Measurement: 

a. Unsure each student achieves every course objective 
before graduation. 

b. Permit earl) identification of students who fail to 
attain ohjectives and need special individual assistance or 
additional training. 

c. Inform students whether or not the) aic attaining the 
objectives, stimulate effective learning, and reinforce 
knowledges and skills. 

d. Fnsun students meet the performance recheck 
requirements, such as typing or code speed specified hv 
using activities, or where a separate agene\ certification is 
required. 

e. Provide data for use in: 

(1) Determining teaching and learning effectiveness 
and improvement of the instructional system. 

(2) Maintaining quality control. 

(3) Fstahhshing a permanent record of each student's 
achievement in the course. 

(4) Selecting students for special recognition 

(5) Validating aptitude test batteries. 



4. Policies: 

a. Before graduation, each student achieves all course 
object! , s as demonstrated by a progress check, com- 
prehensive written test, or performance test and pass all 
sampling written unit, block, and (or) end-of-coursc tests. 
Students may graduate it failure to accomplish ohjccti\cs is 
due to the existence of a training deficiency. Document 
training deficiencies as required in AFR 8-13. ATC Sup I. 

h. Student measurement is not required in orientation, 
symposium, and familiarization type courses. Awaid final 
course grade of satisfactory (S). 

c. Use student measurement in all type 2. 3, and 4 
courses (AFM 50-5) and military training courses except 
those in h, a hove. 

d. Measurement of students attending type 1 and 5 
courses follows the pioeedures specified in the contract and 
appropriate course doemnention or as specified h> the 
agency responsible for establishing the curriculum for 
training. Where possible, measurement follows the intent of 
this regulation. 

c. The procedures for airman basic military training 
courses: 

(1 ) Record written test and performance test grades on 
locally designed foim. 

(2) Recoid performance test grades as satisfactoi) (S) 
or unsatisfactory (V:). 

(3) I se adjectival grades of outstanding (O). sat isf ac- 
tor} (S). or unsatisfactory iU) for bitten tests. 

(4) Record overall grades as outstanding (O), satisfac- 
tory (Si. or unsatisfactory (U). 

f. lime allotted for administration or written tests, test 
critiques, and performance tests is shown in the course 
chart and POI. The time allotted must allow the 
measurement of each student on each ohjective. Estahlish 
special policies and procedures consonant with AFM 50-29, 
as appropriate, for testing Security Assistance Urainiiu 
Program trainees. Frequent measurement is encouraged to 
minimize washhack time and to pro\ ide carl) identification 
of students in need of special individual assistance. 

g. De\elop sampling written unit, block, and (or) end-of- 
course tests to sample the knowledge components of each 
objective assessed during a pi ogress check. The prima:) 
purposes of these tests are to aid in retention, act as a 
qualify control de\iec. and assign grade*. LstaMish .i pass 
or fail point no lower than 60 percent. 

h. When progress checks are not used to assess student 
accomplishment of knowledge oriented objectives. de\clop 
comprehensive written tests to determine student achieve- 
ment. The pass or fail point is established b\ the standard 
stated in each objective. 

i. (trades aie assigned for both written and pci'oMUikc 
tests, (trade written test-, using percentage scoie* Perfor- 
mance tests- may he graded using either the sat^factor.(S) 
or iinsatisfactoryd 1 ) method, oi percentage scores When 
the S-l! method i* used on petformance tests, the final 
course grade is the average of pciccntage stoics aHaiiud on 
written tests When a pciccntage score is used on 
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performance tests, the final course grade is the average of 
the scores obtained on the performance and written tests. 

j. Establish and maintain a documented course 
measurement plan indicating the measurement device used 
to satisfy each objective. 

5. Responsibilities: 

a. ATC.'TTSA establishes basic polic>. provides 
guidance, and maintains surveillance over the student 
measurement program. 

b. Training wings/ schools/ 3785FLDTG provide staff 
supervision and measurement expertise to ensure that 
student measurement as prescribed herein is implemented 
and remains consonant with the intent of this regulation. 

c. Groups and departments implement and supervise the 
student measurement program according to this regulation 
by: 

(!) Establishing the passing standard for each sampl- 
ing written test. Do not set the minimum grade below the 
equivalent of 60 percent. 

(2) Approving progress checks, written and perfor- 
mance tests, und related materials and instructions 
including revisions 

(3) Authorizing establishment and retention of test 
item pools. 

(4) Ensuring that instructors and instructor super- 
visors arc knowledgeable of the procedures for conducting 
the measurement program for their respective courses. 

(5) Providing guidance and measurement expertise to 
individuals involved in preparing draft tests and test 
materials, administering and critiquing progress checks and 
tests, and documenting measurement results. 

(6) Conducting test analysis and revising the measure- 
ment program as required. 

6. General Procedures: 

a. Measurement Plan: 

(\) For each course objective, document the type of 
measurement device (performance test, progress check, or 
comprehensive written test) used to satisfy each objective, 
the test number, the specific measurement itcin(s) covering 
each objective, and the training standard item supported hy 
each objective. 

(2) The measurement plan may be documented on the 
POI. part I of the lesson plan, or on a separate document. 

b. Progress Checks: 

(1) List or reference the objectives on AIC Form 667 
or 667A that are to be assessed through the use of a 
progress check. 

(2) Each student's performance is evaluated bv the 
instructor on each objective that is to be satisfied bv a 
progress check while the student is in the teaching-learning 
environment. The instructor determines if the student 
successfully accomplished the objective based on the 
behavioral outcome stated in the objective applicable 
checklist (ATC Form 98 or locally produced checklist), and 
his or her judgment of the student's performance. For 



objectives that require group activity, the instructor rates 
each member according to the individual's performance 
and participation in the group. 

(3) When a progress check cannot be conducted 
during the block because of equipment malfunction, 
nonavailability, etc, the student may continue in training 
provided the progress check or a performance test on the 
same objective is completed before graduation. When the 
equipment malfunction, nonavailability, etc.. cannot be 
corrected before graduation, document this as a training 
deficiency. See AFR 8-I3/ATC Sup !. 

(4) After the instructor determines that the student has 
satisfactorily attained an objective identified on the 
criterion checklist, an "S % * or percentage score is entered on 
ATC Form 667 o» 667A. Enter "IT for unsatisfactory. 

(5) If. after special individual assistance or practice, a 
student cannot attain a satisfactory rating on a progress 
check, then consider washback or removal from training 
action. 

(6) Retain completed ATC Form 667 or 667A until 
the results are recorded on ATC Form I56. Student Record 
of Training. Retain the ATC Form 667 or 667A completed 
during validation until the next course revision. Destroy ii 
according to AFM 12-50. table 50-1. rule 12. 

(7) Record the overall rating for the block as an "S/" 
"U." or percentage score of the ATC Form 156 or 379. 
Attendance and Rating Record. If a student is removed 
from training, annotate the ATC Form 156 or 379 as 
prescribed in ATCR 52-11. 

c. Performance Tests: 

(1) Test each student on every objective that is 
designated for a performance test. When the objective calls 
for group performance, the instructor rates each member 
according to the individual's pcrformar *c and participation 
in the group. 

(2) Develop a checklist for each objective measured bv 
a performance test using ATC Form 98 or locally approved 
checklist. The checklist should include specified standard^ 
relative to instructor assistance and safety, a breakdown ol 
technical data, an explanation of its use. or anv other 
inlonnation necessary to ensure a high degree of objeeiivitv 
in determining satisfactory performance. 

(3) Develop standardized instructions to thccxainin. 
for administering and critiquing the performance test. Sec 
attachment 3. 

(4) When practical, an instructor other than tin 
individual who taught the class should administer the test 

(5) Inform the students of their grades and critique 
their performance as soon as practical after administration 

(6) Record performance test grades for each objective 
(performance test item) as "S. % * "U." or percentage score on 
ATC Form 98 or locally developed checklist. 

(7) Record the overall rating for the performance tcM 
as %% S." "U. %% or percentage score on the ATC Form 156 o: 
379. A satisfactory performance test rating require- 
successful accomplishment of each objective measured a: 
this test point in the course. 
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(8) Ruain the completed ATC Form 98 or locally 
approved form until results are entered on ATC Form I56 
or 379. Retain the ATC Form 98 accomplished during 
validation until the next course revision. 

(9) Students who fail performance tests arc considered 
for special individual assistance, washback. or removal 
from training. If a student is removed from training, 
annotate the ATC Form 1 56 or 379 as prescribed in ATCR 
52-IK 

d. Written Tests: 
(!) Develop a sampling written test for those objec- 
tives assessed by progress checks. The sampling should 
include every objective when all test versions arc combined. 
No single test version must include every objective. Proper 
rotation of test versions includes all objectives o\cr a period 
of time and permits validation of each test item. I he 
difficulty, complexity, crittcality, and scope of behavior 
specified by the objective is used in determining the number 
of test questions required for each objective. 

(2) If the knowledge objectives have not been satisfied 
through the use of progress checks, construct a comprehen- 
sive written test so as to provide for documentation of the 
attainment of the objectives tested. The number of 
questions relating to a particular objective is determined by 
the objective itself. For example, if an objective includesa 
standard of "4 out of 5." the test must havr five questions 
on that objective and the student must get at leas! lour ol 
these questions correct to demonstrate satisfactory achieve- 
ment of ihe objective. 

(3) Develop enough test versions (minimum of two) 
for eacn measurement point in a course to provide alternate 
tests in the event of a test compromise. The selection of 
questions should not exceed 25 percent of the questions in 
any _othcr Jest version. Do not count scrambled or 
resequcneed items as test version. 

(4) Develop standardi/ed instructions to the examiner 
for administering and critiquing the written test. See 
attachment 4. 

(5) Unsupervised breaks are not taken during the 
administration of written tests. Students ma> leave the 
room only when they have completed their test and on 
submission of their testing materials to the examiner. 

(6) Use technical data whenever it contributes to on- 
the-job realism. 

(7) Tests arc critiqued and students informed ol their 
grade as Mson as practical after administration. Scored 
answer sheets and copies of the test are returned 
temporarily to all students for their rewew during the 
critique (except BM1). lake c^.rv 10 reduce the possibility 
of test compromise. 

(8) Record written test results on AK" l-orm 156 or 
379 as a percentage score. Circle failing semes*. 

(9) When a student receives a failing grade, follow the 
procedures in ATCR 52-26. paragraph Ik. When a student 
passes ;hc ret est. record the minimum passing seme allowed 
by the test as that student's grade and indicate in the 
remarks section of the AK Form 156 the student's actual 



score. Remove a student from training when it becomes 
apparent the stud.nt cannot complete the course. 

(10) Conduct an analysis of test items as soon as 
possible after each test for the first three classes, 200 cases, 
or I vcar, whichever is sooner. After initial validation, 
conduct further test anal} sis on a random basis. The intent 
of test item analysis is to ensure quality control and 
determine the elfectiveuess of instruction, course materials, 
and test items. Place emphasis on "high" and "low* Thigh" 
miss— 50 percent or more of students miss a question and 
-low" miss less than 5 percent of students miss a question) 
miss rates to determine what corrective action, if any, is 
required. Look for trends of "high" and "low** miss items 
over an extended period of time. Document test item 
analysis on ATC Form 068. Test Data, (see attachment I), 
a locally produced form, or a computer product. Maintain 
test analysis until a test is revised, deleted, or would serve 
no further purpose. 

e. End-ot-Course Written Test/Performance 
Recheck: 

(I j Construct end-ol-course tests to check selected 
knowledge oriented objectives for courses where such a test 
is necessary or desirahle. Performance rechecks are 
administered wheie a special lequirement such as a tvping 
or code speed is included in the course, where separate 
agencv certification is required, or when objectives require 
added qualitv control Other ohjectives may be checked on 
a sampling hasis as deemed appropriate Ihe amount of 
time devoted for end-ot-course testing and performance 
rechecks should be consistent with the need for such testing 
and he cost effective. 

(2) Each student is administeied a performance 
recheck on each objective developed for special re- 
quirements and certification. 

(3) Concrete terms (such as words per minute, etc.) 
used lor test grades are recorded m Al C Form 156. item }. 
part V. 

(4) When cnd-ol-cour.se tests are used, each student 
must obtain a passing score on the written test to he 
graduated, l est grades are recorded on A'K" Form 156 oi 
.179. 

(5) Students who tail to satislactorih complete 
performance rechecks oi do not meet cctt ideation or 
special requirements will not he graduated. 

f. Appraisals: 

(!) Use appraisals .is an interim cheek lor letention 
and earl\ identification of students who need special 
individual assistance. 

(2) Recording or maintaining grades and control is 
not required. 

(3) Appraisals ma> he administered without documen- 
tation in the PCM 

7. Control of Testing Materials: 

a. Store test materials in a storage cabinet secured with a 
combination lock, in a sale, or other compatible secure 
area, lest materials maintained in word processing centers 
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on {apes or floppy discs, and tho.se in diaft stages will he 
secured in the .same manner as a finalized test; however, the 
provisions of paragraphs 7c and d helow do not apply. 
Control performance tests and materials only when they 
contain information that could cause a compromise of the 
test. 

b. Restrict access to materials in locked storage to the 
minimum number ofauthorizd personnel necessary. 

c. Maintain ATC Form 1 005. Test Control, when a test 
is removed from a locked storage area for any purpose, and 
ensure pertinent details arc entered in the "issued to" 
columns. The ATC Form I005 is not a controlled form and 
need not be stored in a secured area. 

d. After a test is administered, the test administrator 
examines the test booklets for missing pages, removes 
pencil marks, and returns booklets to test custodian. The 
test administrator certifies to the reusability of the tests by 
signing the test control form. Custodian's signature 
indicates all issued tests have been returned and the current 
inventory count as shown on ATC Form IU04, Test 
Inventory, is accurate. After the last entry, destroy ATC 
Forms I005 immediately according tc AFM I2-50. table 
II-I, rule 4. 

e. Use a separate ATC Farm 1 004 to maintain a running 
inventory of each different controlled test and its related 
test materials. Keep reserve stocks of controlled tests to a 
minimum. Except for course validation file copy, destroy 
outdated controlled tests and related test materials within 
!0 duty days after implementation date of the replacement 
test. Immediately destroy loose pages lelt over alter 
controlled tests are assembled or maintain inventory of 
loose pages on ATC Form I004. Immediately destroy 
closed out ATC Forms I004 according to AFM 1 2-50. table 
II-I. rule 4. f 

f. Suspend ise of a compromised test or one suspected 
of being compromised for a period determined hv training 
development sections or measurement sections. 

g. Complete ATC Form I004 as follows: 

(1) Type or make all entries in ink. 

(2) The master copy of the written test is not included 
in the balance on hand column. The master cop> is 
identified by writing across the top center of the first page 
"Master Copy** with no numerical or alpha idcntilier. 

(3) Maki* a separate and complete entry for each t\pc 
of material used. Where necessary, use more than one line 
in the identification of material and action taken column. 

(4) Custodians Block. Enter statement. "Posted on 
storage cabinet." 

(5) Identilication of Test Materials. Use the following 
code to identify miscellaneous test materials: 

(a) K— test key(s). 



(h) I) test data fonn(s). 

(c) Additional coding at user discretion. 

(6) Transaction Number. Each individual transaction 
is numhered in sequence. 

(7) Balance on Hand. After each transection \ 
completed, enter the new balance in the column provided 
If one series of tests is destroyed and the same ATC Forn 
I004 is used for the subsequent series of tests, start th 
transaction number over at I. 

h. Destroy completed answer sheets for written test 
immediately after results have heen recorded and test ttci; 
analysis is completed, unless required for specific projects 
in this case maintain an inventory on a general purpos 
form. 

i. Control classified tests under the above provisions i: 
addition to those safeguards outlined in AFR 205-I o 
other applicable security directives. 

8. References: 

a. AFR 8-13/ ATC Sup I, Air Force Specialty Trainin; 
Star 4 s rds. 

b. .FM 12-50, Disposition of Air Force Document,! 
tion. 

e. AFM 50-2. Instructional System Development. 

d. AFM 50-5. USAF Formal Schools Catalog. 

e. AFM 50-62, Principles and Techniques of Instruc 
tion. 

f. AFP 50-58, Handbook for Designers of Instruction;! 
Systems, voiuine III. 

g. A'ICR 50-21, Field Training Detachment (HD 
Piograin. 

h. ATCR 52-6, Curricula Documentation. 

i. ATCR 52-11. Student Record of Training. 

j. ATCR 52-26. Student Scheduling and Administni 
tion. 

k. ATCR 53-1. Faculty Boards. 
I. ATCM 52-9, Guide lor Special Training, 
in. AFR 205*1, Information Security Program. 

9. Supplements. Each center and school OPI 
coordinates supplements to this regulation with H(. 
ATC TTSA for approval before publication. 

10, Implementation, Rc\ise each course to comply wit 
the provisions of this icgulation during the next coup* 
revision or within I year of the date of this publicatior 
whichever is sooner. 

11, Forms: 

a. Prescribed. ATC98,667/667A.668. I005.and I004 

b. Adopted. ATC 156 and 379. 
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ATCR 52-3 : 25 July 1984 




THOMAS M .. RYAN, JR.. General. USAF 
Commander 



PHILIP T. ROBERTS. Colonel. USAF 
Director of Administration 



4 Attachments 

L Completion Instructions for ATC Form 668. Test Data 

2. Completion Instructions for ATC forms 667 and 667A 

3. Instructions to the Examiner— Performance Test 

4. Instructions to the Examiner— Written Test 



SUMMARY OF CHANGES 

This regulation clarifies guidance concerning written and performance tests; differentiates between knowledge and 
performance objectives: authorizes the use of progress checks to satisfy objectives; establishes the use or the ATC Form 668 
as optional for test analysis; authorizes the use of a sampling written block tests and end-of-course tests as retention devices; 
adds the requirement to list all objectives assessed by progress checks on the ATC Form 667 or 667A: authorizes the 
conversion of performance test results- to a percentage score and count as part of the overall course grade; adds the 
requirement to have a documented measurement plan; clarifies the test control procedures for tests stored in word 
processing centers on tapes or floppy discs; establishes the requirement to have supplements to this regulation approved by 
this headquarters before publication; and establishes implementation procedures for complying with the provisions of this 
regulation. 
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COMPLETION INSTRUCTIONS FOR ATC FORM 668, TEST DATA 



The numbers below refer to corresponding circled 
numbers in specific locations on ATC Form 668 shown 
in figure AM. 

1. Enter class number and section or shift of the group 
tested. Example: 68I2IIA. For self-paced courses, enter 
date test data was computed. 

2. Enter the number of students in the group tested. Do 
not count or use any answer sheets of retests or foreign 
students. Example: 8. 

3. Enter the last name of instructor that taught the group. 
Example: Rrown. In self-paced or multigroup entry courses 
leave blank. 

4. Enter the course number. Example: 3ABRXXXX. 

5« Enter number of this specific test. Example: W312-1. 

6* When the form has the required number of groups 
recorded in sections I through 5, enter the total number of 
students recorded in sections I through 5 in this space. 
Example: 42. 

7, Draw a heavy line through or block out the correct 
answer for each question on the test. 

8, Using the answer sheets (ATC Form 26 or 26a) record 
the total answers selected in error in the appropriate A, B, 
C D, or E column for each question. Tally may be 
accomplished on a separate sheet. Destroy tally sheets 
immediately after posting totals on ATC Form 668. 

9, In the appropriate "Number Wrong Items" column 
enter the to; al missed by the entire group for each question. 

10, When the form has the required number of groups 
recorded in the "Number Wrong Items" columns, enter the 
total of the misses for each question in the "Total Wron>>" 
column. 



11* Enter the difference between the"Numberof Cases"(6 
and the "Total Wrong" (I0) column in the "Total Right" 
column. ' 

12. Enter sell the obtainable percentage grades from I0( 
down to below the passing point. 

13. Indicate in the proper columns (l through 5) tin 
number of students in each class attaining each grade. Fot 
self-paced and multigroup entry courses record tin 
accumulated number of cases attaining each grade. 

14. When the form has the required number of group? 
recorded, enter the total number of students attaining each 
recorded grade. 

15. A heavy line is drs^n through this section just undei 
the minimum passing grade to make the grade distributioi 
more meaningful. 

16, 17, and 18. Obtain the class average for each group. !« 
in group I, eight students took the 40 questions test there i? 
a total of 320 answers. By adding the numbers missed ir 
column I under the "Number Wrong Items" section a tota 
of 73 misses is counted, (1 6). By subtracting the number ol 
questions missed (73) from the total possible answers (320) 
the result is the total number of correct answers (247) 
which when divided by the total possible (320) gives tin 
class average— 77.2, (1 7). Use the same method to obtair 
the average difficulty of the completed ATC Form 668 
(!8). 

19. Fntcr the test item number. When a test consists ol 
more than 50 questions, simply continue the count on 
another ATC Form 668. 



NOTE: Sir .sample completed ATC Form 668 in f^urt 



62 



ERLC 



ATCR 52-3 Attachment 1 25 July 1982 



U7 
IB. 





1 


a 


3 


4 


5 


course nukier 


TtST AVERAGE 


CLASS NUMOE* 












NUMtCR OF STUOKMTS 












TUT NUMBER 


NUMBER Or CASES 


no or instructors 













ITEM 
NO 



10. 



11 



1? 



11 



1L 



JLi 



16. 



11 



21 



-21 



24 



15. 



26 



11 



18. 



29 



IS. 



11 



33 



2k 



38 



ANSWtftS SELECTKO W EMO* 



ft 



TT 



f-r-i-r 



-i — i — i — r 



rr 



i 



i 



if 



CLASS AVERAGE 



1 



T i-H 
-t-1 I 



: i 
FT" 

Hi 



n 



Tut al 



132 



TOTAL 



Itlfhf 



Specific CfW« 



2£ 



1 2 3 4 3 



TOTAL 



-1- 



t ATC /^i* 6^8 rrevious edition *«at nt- utei« 
Floure AM. Annotated ATC Form 668 t 



TEST DATA 



63 



76 



ATCR 52-3 Attachmtnt 1 25 July 1962 

— rr 



CLAM NUMRER 



tip Of INSTRUCTORS 



N UM1ERQF STUOENTS^ I g \JQ\_ 1^ 1 

EVALUATION OF SPECIFIC ITEMS 



J J COURSE NUMRER 

wee**** 



TEST NUMRER 



ITEM 



AW SWIM SELECT! 0 IN IMOft 



Wr*D| \\*m% 



TOTAL 



TEST AVERAGE 

4gy 



NUMRER OF CASCS 

2^ 



EVALUATION OF ENTIRE TEST 



TOTAL 



CLASS AVERAGE 



TOTAL 





' A 








c 






0 


t 


i 




3 


4 


s 








1 


3 


3 


4 


s 




1 
















/ 


V 


2 


s 












L 


1 1 


/ 


V 


2 


< 


u 






.00 














. o 








/ — 










/ 










/ 


/ 






n 






ICOSTJBR 


!■■ 






3 




I / 


I 










1 


2 


,5 


1 


z 


x 


















II 


II 






4 


/ 


f f 






/ 










/ 


Z 














-/ 


/ 


■ 




I 


l« 




i 


c 


7 


// 




93 














5 




i J 
f I 


/ 
/ 




/ 


/ 


i 
/ 


/ 












J 




i 
( 




( 


/ 


/ 






/ 


,? 


H 


3 






^7 


a/) 








/ 




I 


6 
































■1 


/ 




/ 






f 


/ 




J- 




/ 


A 






/ 










J 


7 


X 








2 


z 
















/ 


























V 


/? 














/ 


3 




/ 




/ 




J 


/ 




i 


1 




x 












y 






3 








X 


8 








m 1 


2 


3 










t 




i 


+7 
I 


2 


/ 


V ' 


? / 


z 




i 


I 


,? 




f 


? 


< 


Iff 


// 












2 


S 


9 










/ 




/ 
















/ 














/ 


o 


j 


0 


/ 


/ 


s 


,1? 


fo 




% 




f 




7 


10 




I 


/ 




/ 




2 


« 












2 








/ 


/ 






; 


j 


4 


<: 


3 






jy 


7f 






/ 








11 




1 






i 


1 




/ 






2. 




i 
















/ 






J 




I 


y 


i* 


£p 


7f 




J 






( 


* 


12 








f 








/ 














/ 
















0 


J 


(9 






y 


r?f 


7.3 




/ 








t 
/ 


13 


% 










/ 


) 
















/ 




/ 










2 




/ 


/ 


J 






J3 




/ 




- 






/ 


U 










i 




/ 










/ 






J 














/ 




.? 


0 


X 


/ 












J 


1 




* 1 


15 


I 


f 


- 


/ 


i 






















x 
















2 


2 


c 




c 




€& 














16 


I 


f / 




















2 














/ 










S 




X 


1 


IO 














X 


j i 






3 












( 


















1 


? 


/ 








il7 




















i 




























1 


0 


c 


o 


/ 


f/ 






I 


( 


( 






18 


I t 






f — 










/ 


/ 






j 




I 




/ 




/ 


2 




/ 




( 


2 


/ 


3 


ft / 




Si 








/ 


/ 


2 i 


1$ 








- / 


f 


I 


/ 




/ 


2 


1 


1 




/ 


/ 


/ 


2 








/ 




i 


1_ 


( 




/ 




/? 






y 


I 






S 


20 




/ 










/ 










1 




















/ 


a 


Q 




/ 


y 


















21 








3/ 


2 






/ 




/ 


2 




i 




/ 


2 


i 












§ 


f 


( 


/ 


/ 


// 




SQ 




/ 






1 










2 














i 




I 






/ 




1 


/ 








/ 


i 




j 


< 


7 

dL 




f i y 


V? 














33 


I 
























i 
















9 






Q 


A 


* 


/ 


jr 


wfD , 










i 




/ 




















I 






















2. 


/ 










L 






f— ~ 








/ 


<*. 












1 






















0 






jt 
















i 




/ 








































/ 












i— 




25 




i 


/ 


/ 

/ 


I 
i 


/ 


JL 




3 


1 


1 


I 






•2 


JL 


7-1 


f 










f 




y 


7 




If 


7 7 
















26 


j 
/ 


I 














2 


1 










/ 














z 






j 


f 






J?7 








/ 






/ 














9 




7 
/ 
























1 






< 




Z 




T 


















1 




h 






/ 


i 




1 










- y 

# 


2 


/ 


1 




















28 


/ 






L 




/ 


















/ 

w 


/ 


1 


/ 


/ 








J 
/ 




• 

.r 


4/ 

? 


> 


J.Y 


























J 












\ 






f 




/ 




i 

L. _ 






j 


2 






/£ 


















36' 












y 
/ 


4/ 




/ 








i 


A 












/ 


3 


/ 










/r 
































l 




/ 


2 








y 




















31 


i 

m 


r 




/ 


? 


y 


J- 




2 




f 


t 




» 






/ 














/ 


V 


i 




















32 






- j 


1 i in 










/ 








1 




/ 




V 










/ 


y 


3 




y 


\ 


lY 


















33 


4 


r 
















1 


1 




1 




/ 


/ 


I 






/ 






i5 


7 


,3 








Z'' 
















3A 




2 


i 




; 




1 






I 




A 




















J 


2 


1 


3. 


y 




AT 


















35 






j 








I 




1 




V 


/ 










X 












1 


0. 


?| 


3 




JLi- 


















36 


— 4- 






- / 










I 




.? 








/ 


/ 


/ 










1 


A 


✓ 


j? 


AJ 


















3? 




U 


i 


/ 






I 








/ 


t 














/ 


/ 




? 






i 




2 


/? 


















38 
















• 








X 






















0 


0 


0 




0 




















39 






















/ 


/ 










/ 










J 






/ 


,3 


3 




















40 






















/ 












/ 










I 


0 


/ 


I 


/ 


I 


































































































































































































































































































































































































































































































































































































































































































































































J 





















<rs;7 y /// 



70.0 



jatc ;o;- m« 



PREVIOUS EDITION MAY RE U8E0 UNTIL STOCK IS EXHAUSTED. 



fEST DATA 



q FIflurt A1-2. Samplt Compltttd ATC Form fitt 



64 



77 



10 



ATCR 52-3 Attachment 2 25 July 1982 



COMPLETION INSTRUCTIONS FOR ATC FORMS 667 AND 667A 



1. Type or make all entries in ink. 

2* Before students enter a block of instruction, or 
immediately thereafter, prepare a criterion checklist 
consisting 6f as many ATC Forms 667 or 667 A as required. 



3. Enter the PO! paragraph reference or the objectives 
which is measured by progress checks on ATC Form 667 or 
667A. 



4. Use ATC Form 667A for FTD courses and courses 
without a POl to record objectives. 



5. Before adr nistering a progress check, tell the student 
what is expected and explain that the rating received 
depends on nis or her performance. 

6. Completion of blocks on ATC form 667: 

a. Course number, block number, instructor's name, 
grade, and initials are self-explanatory. 

b. Under student's name enter last name and initials of 
each student attending the block of instruction. 

c. Opposite the PO! paragraph reference on ATC Form 
667, enter an **S" for satisfactory, a "U" for unsatisfactory, 
or percentage score. 
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INSTRUCTIONS TO THE EXAMINER— PERFORMANCE TEST 

EXAMPLE 



DEPARTMENT OF THE AIR FORCE 
USAF Technical Training School 



TEST II 



MEASUREMENT 



COURSE:. All courses 

SUBJECT: Instructions to the Examiner (Performance Test) 

OBJECTIVE: To provide standardized instruction for test administration and space to include a list of the tech dai 
that will be used 

INSTRUCTIONS: 

1. When a controlled performance test is administered or removed from the locked storage area for any purpose, ensui 
that the date, time, number sequence of tests issued, and the signature of the person to whom the test is issued are entered o 
the A TC Form I005, Test Control. 

2. Unsupervised breaks a;e not permitted "during the performance test. 

3. Keep the testing area free of distractions as much as possible during examination. Administer makeup te*t in a quie 
area o: lab atmosphere free from all interferences or distractions. 

4. Student notes and materials should be covered or kept in a folder during the examination period. 

5 Use of tech data during test periods is permitted and encouraged when it contributes to on-the-job realism provided it 
use does not compromise the test. 

6. Scratch paper should be furnished by the instructor if needed and collected with the answer sheets and examination:* 

7. Clearly indicate time available for taking examinations to the students and what is required for satisfactor 
accomplishment. 

8. Advise students if they are authorized to ask questions or receive instructor assists during the testing period. 

9. If group testing is required, ensure each student understands their role(s) as a group participant. 

10. Instruct students not to mark test booklets. 

11. Tests are critiqued and each student informed of their score as soon as feasible after administratis. 

12. Copies of the test should be returned temporarily to the students for r^eir review during the critique. Take care tc 
reduce the possibility of test compromise, therefore, during the critique period use two instructors if possible. Students an 
not permitted to use any written materials or recording equipment and arc not permitted to leave the room. 

13. Instructors check test booklets for markings before returning to the storage area. 

14. The person actually returning the examinations to the custodian certifies the examinations arc clean, even though tha 
person may have obtained custody of the examinations through reissue. 

15. List technical data required during this test administration. 
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INSTRUCTIONS TO THE EXAMINER— WRITTEN TEST 
EXAMPLE 



DEPARTMENT OF THE AIR FORCE 
USAF Technical Training School 



TEST ID 

MEASUREMENT 



COURSE: All courses 

SUBJECT: Instructions 4o the Examiner (Written Test) 

OBJECTIVE: To provide Mandated instruction for test administration and space to include a list of the tech data that 
will be used 

INSTRUCTIONS: 

1. Whe»" a lest is administered or removed from the locked storage area for any purpose, ensure that the date, time. 

?^ I scc ^ ucncc of tcsts issucd . and the signature of the person to whom the test is issued are entered on the ATC Fori* 
I005, Test Control. 

2. Unsupervised breaks are not permitted during measurement test. 

3. Keep the classroom free of distractions surh as charts, diagrams, and cutaways during examination. Administer 
makeup test in a quiet classroom or lab atmosphere free from all interferences or distractions. 

4. Student notes and materials should be covered or kept in a bluer during the examination period. 

5. Use of tech data during test periods is permitted and encouraged when it contributes to on-the-job realism pro\ided its 
use does not compromise the test. 

6. If possible, scat the students in alternate scats. In some classrooms it may be possihle to move the chairs farther apart. 

7. Scratch paper should be furnished by the instructor if needed and collected with the answer sheets and examinations. 

8. Clearly indicate time available for taking examinations to the students and minimum passing grade. 

9. On multiple choice type questions instruct the students to select the best choice of the answers listed. 

10. Advise the students to answer the questions they are confident of first, saving the more difficult questions until last. 

11. Instruct students to fill in all applicable headings on the answer sheet. 

12. Instruct students not to mark test booklets. 

13. Tests are critiqued and each student informed of their scores as soon as feasible after administration. 

14. Scored answer sheets and copies of the test should be returned temporarily to the students for their review during the 
critique. Take care to reduce the possibility of test compromise; therefore, during the critique period use two instructors if 
possible. Students are not permitted to use any written materials or recording equipment and are not permitted to leave the 
room. r 

15. Review test analysis data. 

16. Instructors check test hooklets for narking* before returning to the storage area. 

17. The person actually returning the examinations to the custodian certifies the examinations are clean, even though that 
person may have obtained custody of the examinations through reissue. 

18. Annotate a copy of these instructions lor each written test and issued with the test. 

19. List technical data required during this test administration. 
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(ATC FORM 156) 
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APPENDIX D: AVERAGE DIFFICULTY 



Average 

# Title Rating 

1 Speech 2.4 

2 Algebra 4.1 

3 Electronics 3*4 

4 Biology 4.0 

5 Accounting 3.0 

6 Business Math 2.6 

7 Chemistry 4.6 

8 Commercial Art 2.1 

9 Mechanical Drawing 2,4 

10 General Science 3.0 
.11 Geometry 4.2 

12 Journal uai 3.4 

13 Zoology 3.8 

14 Physics 4.9 

15 Psychology 3.7 

16 Projection Equipment 1.6 

17 Radio Repair 2.5 

18 Trigonometry 4.4 

19 Use of Blueprints 2.8 

20 Anatomy 3.9 

21 Auto Repair 2.6 



RATINGS BY HIGH SCHOOL COURSE 

Average 



# Title Rating 

22 Hydrology 3.7 

23 Hygiene 1.5 

24 English 3.8 

25 Industrial Arts 1.8 

26 Logarithms 4.0 

27 General Business 2.4 

28 Business 3.0 

Administration 

29 Driver Training 1.5 

30 Geography 3.1 

31 Management 2.9 

32 Marketing 2.7 

33 Mechanical Theory 3.4 

34 Merchandising 2.2 

35 Home Economics 1.7 

36 Business Machine 1.5 

Operator 

37 Statistics 4.2 

38 General Math 2.6 

39 bhop Math 2.1 

40 Vocational Guidance 2.4 

41 Music 2.2 

42 Agriculture 2.3 
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APPENDIX £ : COURSE CONTENT VARIABLES 8Y COURSE 



Chanute Technical Training Center 
Aircraft Environmental Systems Mech. 
Aircraft Fuel Systems Mech. 
Airframe Repair Tiec. 
Missile Maintenance Spec. 
Special Vehicle Mech. 
Fire Protection Spec. 
Aircraft Electrical System Spec. 
Jet Engine Mech. 

Sheppard Technical Training Center 

Aircraft Loadmaster 

Telephone Switching Spec. 

Cable Splicing Spec. 

Tactical Aircraft Maintenance Spec. 

Electrician 

Carpentry Spec. 

Financial Services Spec. 

Medical Services Spec. 

Surgical Services Spec. 

Medical Assistant Spec. 

Physical Therapy Spec. 

Dental Assistant Spec. 

Financial Management Spec. 

Medical Laboratory Spec. 
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APPENDIX E : ( Cont 1 nued) 



Course Title 



tachland Military Training Center 
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Security Spec* 


81130 
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11.1 


2.4 


20.1 


.35 


Law tnf orcement Spec* 


81132 
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254 
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13.0 
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20.6 


.11 


Lowry Technical Training Center 


















Continuous Photoprocessing 


23330 
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24.0 


.32 


Instrumentation Mech. 


31633 
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40.2 
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Avionic Sensor Systems Mech. 
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Precision Measuring Equipment Spec. 
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Aerospace Ground Equipment Spec* 
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Computerized Test Station Spec. 


32634 


M 


712 


6 
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Armament Systems Spec. 


47230 


V 


412 


6 


6.1 


2.8 


MD 


.40 



88 



.54 


.22 


12.0 


1 


7767 


92 


10.6 


20 


.08 


1. 


.22 


.49 


14.9 


0 


1982 


94 


9.7 


18 


.08 


1. 


.48 


.24 


8.0 


0 


132 


70 


11.4 


40 


MD b 


21 


.34 


.48 


4.4 


0 


188 


116 


11.6 


163 


.10 


j. 


.58 


.53 


6.1 


1 


61 


109 


11.6 


112 


.10 


3. 


.33 


.20 


11.3 


0 


700 


127 


12.0 


110 


.20 


4. 


.28 


.45 


4.2 


0 


28 


137 


12.2 


187 


.10 


3. 


.80 


.54 


4.3 


1 


60 


MD 


11.0 


166 


.00 


3. 


.18 


.39 


5.5 


1 


63 


123 


11.3 


36 


.10 


3. 


.38 


.35 


7.9 


0 


1500 


114 


11.2 


60 


.08 


2. 


.22 


.39 


4.5 


0 


911 


70 


10.3 


34 


.05 


1. 


.51 


.25 


5.4 


0 


260 


134 


11.4 


76 


.07 


3. 



89 



APPENOIX E : (Concluded) 



Course Title 

Keesler Technical Training Center 

Command and Control Spec. 

Wideband Communication Equipment Spec. 

Electronic Computer Spec. 

Telecommunications Control Spec. 

Airborne Warning Radar Spec. 

Electronic Warfare Systems Spec. 

Computer Spec. 

Administration Spec. 

Personnel Spec. 

Persor ! Affairs Spec. 

Ground Radio Operator 

Aircraft Warning Radar Spec. 

Navigation Systems Spec. 

Computer Programming Spec. 
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*M - Course included in model-development sample; V 
b M0 - Missing data. 



Course included in cross-validation sample. 
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APPENDIX F: STUDENT INPUT VARIABLE MEANS AND STANDARD DEVIATIONS BY COURSE 



Educa- 

Speclalty tlonal 
code Sample Selector AI Hgcnanicaj Administrative General Electronics level 



Chanute Technical Training Center 
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SD 


X 


SD 


X 


SD 


X 


SD 


X 


SD 
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SD 


Aircraft Enviromental Sys. Specialist 


42331 


M 


60.1 


19.3 


59.7 


19.3 


55.0 


19.8 


57.5 


16.7 


58.0 


16.6 


2.1 
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Aircraft Fuel System Mechanic 


42333 


M 


61.4 


17.5 


60.1 


17.9 


60.0 


18.9 


59.6 


17.0 


59.6 


17.4 


2.1 
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Airframe Repair Specialist 


42735 
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64.5 


17.9 


63.9 


18.4 


49.5 


21.1 


55.7 


13.2 


59.7 


15.1 
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Missile Maintenance Specialist 
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M 


62.7 


16.0 


62.7 


16.0 


56,1 


19.3 


56.9 


15.0 


60.2 
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.2 


Special Vehicle Mechanic 


47231 


M 


68.6 


16.8 


66.7 


17.1 


55., 2 


18.3 


58.7 


17.8 


61.2 
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Fire Protection Specialist 


57130 


M 


62.9 


15.5 


54.7 


20.7 


56.1 


18.9 


62.5 


15.3 
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Aircraft Electrical System Specialist 


42330 


V 


74.6 


14.1 


65.8 


20.4 


64.3 


19.2 


68.7 


17.3 


74.0 


14.8 


2.1 


.4 


Jet Engine Mechanic 


42632 


V 


64.5 


18.6 


64.5 


18.7 


53.6 


19.7 


59.2 


15.5 


60.3 


16.9 


2.1 


.4 


Sheppard Technical Training Center 






























Aircraft Loadmaster 


11430 


M 


61.7 


18.6 


59.6 


19.0 


61.1 


19.7 


61.5 


15.0 


61.9 


15.4 


2.1 


.3 


Telephone Switching Specialist 


36231 


M 


72.7 


15.0 


64.7 


21.2 


63.2 


18.8 


68.4 


16.3 


72.5 


15.0 


2.1 


.3 


Cable Splicing Specialist 
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58.2 


15.4 


58.2 


15.9 
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17.3 


59.8 
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Tactical Aircraft Mclnt. Specialist 


43131 


M 


61.2 


20.1 


61.2 


20.1 


48.5 


19.8 


53.4 


17.9 


57.8 


18.2 


2.1 


.3 


Electrician 


54230 


M 


73.9 


12.5 


63.1 


20.3 


60.7 


20.1 


71.9 


15.3 


73.9 


12.5 


2.1 


.3 


Carpentry Specialist 


55230 


A 


62.2 


15.7 


62.2 


15.7 


54.1 


19.3 


58.6 


14.9 


61.7 


16.6 


2.0 


.2 


Financial Services Specialist 


67232 


M 


81.1 


11.1 


44.5 


22.2 


81.8 


10.1 


67.3 


17.5 


59.6 


18.7 


2.2 


.5 


Medical Services Specialist 


90230 


M 


66.9 


14.5 


49.2 


22.3 


62.0 


18.4 


66.7 


14.0 


61.9 


17.9 


2.1 


.3 


Surgical Services Specialist 


90232 


M 


67.6 


15.9 


51.1 


22.4 


65.4 


19.6 


67.6 


15.9 


63.1 


18.0 


2.2 


.5 


Medical Administrative Specialist 


90630 


M 


65.8 


14.9 


42.2 


19.7 


66.0 


19.0 


65.7 


14.9 


57.9 


17.3 


2.1 


.4 


Physical Therapist Specialist 


91330 


M 


75.3 


12.0 


52.0 


23.8 


66.0 


18.0 


75.3 


12.0 


67.5 


14.7 


2.2 


.5 


Dental Assistant Specialist 


98130 


M 


6.5.7 


14.1 


45.1 


22.4 


61.1 


19.2 


65.5 


14.3 


59.2 


16.5 


2.2 


.5 


Financial Management Specialist 


67231 


V 


82.9 


10.1 


50.5 


26.0 


82.9 


10.1 


74.8 


18.4 


68.6 


19.7 


2.4 


.4 


Medical Laboratory Specialist 


92430 


V 


73.5 


14.1 


52.2 


23.2 


69.5 


17.8 


73.5 


14.2 


69.4 


17.1 


2.5 


.6 


Lackland Technical Training Center 






























Security Specialist 


81130 


M 


59.6 


14.9 


52.2 


20.2 


57.2 


78.7 


59.9 


14.9 


58.0 


16.7 


2.0 


.3 


Law Enforcement Specialist 


81132 


M 


64.6 


16.0 


52.1 


21.8 


59.6 


20.0 


64.5 


16.0 


59.6 


18.3 


2.1 


.4 
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APPENDIX F : (Continued) 



Specialty 
code Sample 



Academic Simple 
motivation Interest 



Preference Reading 
Interest level 



Educational 
preparation 



Age 



Chanute Technical Training Center 

Aircraft Environmental Sys. Mechanic 42331 

Aircraft Fuel Systems Mechanic 42333 

Airframe Repair Specialist 42735 

M 1 sslle Mai ntenanace Sped al 1 st 44330 

Special Vehicle Mechanic 47231 

Fire Protection Specialist 57130 
Aircraft Electrical System Specialist 42330 

Jet Engine Mechanic 42632 

Sh eppard Technical Training Center . 

Aircraft Loadmaster 11430 

Telephone Switching Specialist 36231 

Cable Splicing Specialist 36131 
Tactical Aircraft Maintenance Specialist 43131 

Electrician 54230 

Carpentry Specialist 55230 

Financial Services Specialist 67232 

Medical Services Specialist 90230 

Surgical Services Specialist 90232 

Medical Administrative Specialist 90630 

Physlclal Therapist Specialist 91330 

Dental Assistant Specialist 98130 

Financial Management Specialist 67231 

Medical Laboratory Specialist 92430 

Lackland Military Training Center 

Security Specialist 81130 

Law Enforcement Specialist 81132 
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M 
M 
M 
M 

M 

M 
M 
M 
M 
M 
M 
V 
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SO 
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SD 
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SD 


X 


SD 


X 


SD 


X 


SD 


36.6 


14.6 


1.3 


.48 


3.9 


1.00 


11.1 


1.0 


1.8 


.44 


20.2 


2.0 


M 


36.1 


13.3 


1.5 


.50 


4.2 


.97 


11.3 


1.1 


1.7 


.61 


19,7 


1.9 


M 


35.7 


12.6 


1.4 


.49 


4.3 


.95 


11.0 


1.0 


2.2 


1.00 


20.0 


2.0 


M 


34.9 


12.9 


1.9 


.29 


5.0 


.00 


11.0 


1.0 


1.1 


.35 


19.7 


1.6 


M 


36.1 


13.5 


1.3 


.48 


4.1 


.99 


11.1 


1.0 


1.3 


.49 


20.1 


2.1 


M 


34.8 


11.7 


1.4 


.49 


4.5 


.86 


11.2 


1.0 


2.0 


.61 


19.7 


1.8 


V 


44.1 


15.1 


lc5 


.50 


4.6 


.77 


11.6 


1.0 


.0 


.00 


20.2 


2.0 


V 


35.3 


10.5 


1.5 


.30 


4.7 


.68 


11.3 


1.2 


1.4 


.49 


19.8 


2.0 



35.7 

41.8 

37.4 

36-8 

41.5 

33.4 

40.0 

37.6 

38.0 

37.7 

38.7 

38.1 

43.2 

46.9 



10.8 

13.4 

l!.5 

13.3 

13.3 

13.6 

12.5 

13.1 

12.3 

11.5 

11.1 

11.3 

13.7 

11.3 



1.3 
1 3 
1.5 
1.7 
1.0 
1.2 
1.3 
1.3 
1.4 
1.4 
1.0 
1.4 
1.6 
1.5 



.49 
.49 
.50 
.44 
.28 
.43 
.47 
.48 
.49 
.49 
.50 
.49 
.48 
.50 



35.4 12.4 
35.7 11.4 



'1.7 .40 
1.3 .49 



5.0 
4.3 
5.0 
4.8 
4.3 
3.9 
4.2 
4.2 
5.0 
4.4 
3.6 
4.4 
4.6 
4.5 



4.8 
4.5 



.00 
.93 
.00 
.57 
.96 
1.00 
.97 
.97 
.00 
.89 
.93 
.91 
.76 
.86 



.58 
.85 



11.6 
11.5 
11.0 
11.1 
11.5 
11.0 
11.6 
11.4 
11.6 
11.4 
11.9 
11.5 
11.7 
11.8 



.84 
.92 
.96 

1.00 

1.00 
.96 
.98 
.94 

1.00 
.87 
.85 
.94 

1.00 
.98 



11.2 .98 
11.4 1.00 



1.7 
1.3 
1.0 
1.4 
1.1 
1.3 
1.1 
3.1 
3.4 
1.2 
1.9 
1.4 
1.2 
3.7 



1.0 
1.0 



.42 
.48 
.24 
.67 
.39 
.47 
.31 
1.00 
.99 
.40 
.64 
.49 
.42 
.86 



.00 
.00 



21.5 
19.8 
19.8 
20.3 
19.6 
19.5 
20.1 
20.0 
20.3 
20.1 
21.0 
20.3 
20.8 
21.0 



3.0 

1.7 

1.7 

1.8 

1.9 

1.6 

2.2 

2.2 

2.2 

2.4 

2.5 

5.6 

2.1 

2.5 



19.3 1.7 
19.7 2.1 



APPENDIX F : (Continued) 



Specialty 
code Sample 



Selector AI Mechanical Administrative General 



Educa- 
tional 

Electronics level 



Lowry Technical Training Center 






X 


SD 


X 


SD 


X 


SD 


X 


SD 


X 


SD 


X 


S 


Continuous Photoprocessing Specialist 


23330 


M 


""66.6 


15.1 


50.9 


21.5 


62.5 


20.1 


66.6 


15.1 


62.8 


17.6 


2.3 


.6 


Instrumentation Mechanic 


31633 


M 


78.7 


9.2 


71.3 


17.1 


65.0 


22.2 


73.3 


16.1 


76.7 


9.2 


2.1 


.3 


Avionic Sensor Systems Specialist 


32232 


M 


79.6 


7.8 


65.4 


22.8 


66.0 


16.3 


72.2 


13.4 


79.3 


7.4 


2.1 


.4 


Precision Measuring Equip* Specialist 


32430 


M 


82.6 


10.0 


73.3 


17.4 


72.5 


17.4 


80.2 


15.5 


82.6 


10.0 


2.2 


.4 


Aerospace Ground Equipment Specialist 


32630 


M 


78.9 


8.8 


71.2 


18.5 


65.4 


19.9 


78.1 


11.0 


78.9 


8.8 


2.0 


.2 


Computerized Test Station Specialist 


32634 


M 


74.0 


9.1 


65.5 


11.9 


71.2 


17.2 


72.0 


13.6 


74.0 


9.1 


2.1 


.3 


Attack Control Systems Specialist 


32636 


M 


76.9 


11.2 


71.7 


16.3 


62.0 


19.8 


71.9 


14.3 


77.6 


10.2 


2.2 


.5 


Munition Systems Specialist 


46130 


M 


62.0 


14.0 


59.9 


15.7 


56.0 


20.2 


59.0 


15.0 


60.6 


15.1 


2.1 


.3 


Material Facilities Specialist 


64531 


M 


61.1 


16.4 


45.3 


21.3 


59.9 


19.0 


60.4 


16.3 


58.0 


17.6 


2.1 


.4 


Armament Systems Specialist 


46230 


V 


59.1 


17.9 


58.8 


18.3 


51.9 


20.9 


57.7 


15.2 


56.9 


14.6 


2.0 


.3 



Keesler Technical Training Center 

Command and Control Specialist 27430 M 63.3 16.9 50.0 23.2 65.0 19.4 62.9 15.6 54.2 22.0 2.4 .6 

Wideband Comm. Equipment Specialist 30430 M 77.5 8.4 68.8 18.5 62.8 16.8 72.6 13.2 77.5 8.4 2.2 .4 

Electronic Computer Specialist 30534 M 80.5 9.9 69.8 17.9 73.3 16.5 79.2 14.0 80.4 9.4 2.3 .5 

Telecommunications Control Specialist 30730 M 76.7 8.9 61.9 18.8 70.0 15.9 71.8 14.4 76.8 8.8 2.2 .5 

Airborne Warning Radar Specialist 32832 M 77.5 10.1 71.0 14.8 59.5 15.4 75.5 11.3 77.5 9.6 2.2 .4 

Electronic Warfare Systems Specialist 32833 M 81.4 8.9 73.0 16.3 65.8 17.9 76.4 13.4 81.1 9.7 2.2 .4 

Computer Specialist 61130 M 83.8 11.2 62.4 21.9 76.3 16.2 83.6 11.6 77.8 15.5 2.2 .5 

Administration Specialist 70230 M 68.3 15.8 38.5 19.8 68.3 15.8 57.3 15.0 51.6 17.2 2.1 .4 

Personnel Specialist 73230 M 72.3 13.2 39.5 22.0 72.3 13.2 58.2 7.2 52.2 20.0 2.2 .5 

Personal Affairs Specialist 73231 M 75.8 15.5 37.2 22.5 75.8 15.5 61.2 19.8 52.4 22.7 2.2 .4 

Ground Radio Operator 29333 V 71.5 12.8 42.9 21.4 71.5 12.8 59.8 16.6 56.4 17.9 2.1 .4 

Aircraft Warning Radar Specialist 30332 V 81.0 6.9 73.1 15.5 65.0 18.7 76.1 11.9 81.0 6.9 2.2 .5 

Navigation Systems Specialist 32831 V 79.6 8.4 72.0 15.9 63.5 18.4 76.7 12.5 79.7 8.1 2.2 .4 

Computer Programming Specialist 51131 V 84.0 12.7 70.3 18.1 78.3 12.5 86.6 9.2 84.5 10.1 2.4 .5 
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APPENDIX F : (Concluded) 



Specialty 
code Sample 



Academic Simple Preference Reading Educational 
motivation Interest Interest level preparation Age 



Lowry Technical Training Center 
Continuous Photoprocesslng Specialist 
Instrumentation Mechanic 
Avlonlc Sensor Systems Specialist 
Precision Measuring Equip. Specialist 
Aerospace Ground Equipment Specialist 
Computerized Test Station Specialist 
Attack Control Systems Specialist 
Munition Systems Specialist 
Material Facilities Specialist 
Armament Systems Specialist 

Keesler Technical Training Center 
Command and Control Specialist 
Wideband Conn. Equipment Specialist 
Electronic Computer Specialist 
Telecommunications Control Specialist 
Airborne Warning Radar Specialist 
Electronic Warfare Systems Specialist 
Computer Specialist 
Administration Specialist 
Personnel Specialist 
Personal Affairs Specialist 
Ground Radio Operator 
Aircraft Warning Radar Specialist 
Navigation Systems Specialist 

Computer Programming Specialist 

Note. The selector AI and the ASVAB 

different sources. This accounts for ml 







Y 
A 


cn 
ou 


V 
A 


SD 


X 


SD 


X 


SD 


X 


SD 


X 


SD 


23330 


M 


38.9 


13.0 


1.4 


.49 


5.0 


.00 


11.5 


1.00 


1.4 


.59 


20.1 


2.3 


31633 


M 


45.2 


14.5 


1.2 


.41 


5.0 


.00 


11.7 


.91 


1.3 


.49 


20.3 


2.3 


32232 


M 


45.3 


14.2 


1.3 


.47 


4.8 


.53 


11.7 


.81 


1.9 


.83 


20.8 


1.8 


32430 


M 


47.3 


12.7 


1.6 


.47 


4.7 


.60 


12,1 


.87 


1.9 


.87 


21.1 


2.1 


32630 


M 


42.5 


13.9 


1.4 


.49 


4.4 


.92 


12.0 


.65 


1.7 


.83 


20.1 


2.1 


32634 


M 


45.9 


10.8 


1.0 


.22 


3.5 


.91 


11.7 


1.00 


1.5 


.79 


21.2 


2.9 


32636 


M 


43.9 


12.4 


1.1 


.31 


3.6 


.96 


12.0 


.90 


1.3 


.46 


21.3 


2.7 


46130 


M 


38.2 


12.0 


1.4 


.49 


4.6 


.77 


11.1 


1.00 


.00 


.00 


19.6 


1.9 


64531 


M 


37.2 


11.2 


1.4 


.50 


4.2 


.97 


11.0 


1.10 


1.0 


.00 


19.7 


1.8 


46230 


V 


37.1 


12.1 


1.5 


.50 


5.0 


.00 


11.4 


1.30 


1.3 


.49 


19.7 


1.7 



27430 
30430 
30534 
30730 
32832 
32833 
61130 
70230 
73230 
73231 
29333 
30332 
32831 
51131 



N 
N 
N 
N 
N 
N 
N 
M 
N 
N 
N 
V 
V 
V 



34.0 
41.7 
45.0 
45.0 
40.9 
48.1 
44.4 
37.3 
37.2 
37.0 
39.0 
45.2 
44.5 
45.7 



10.9 
13.0 
14.7 
13.7 
13.3 
15.0 
14.7 
13.0 
12.3 
13.4 
15.5 
13.7 
11.0 
13.5 



1.4 
1.6 
1.4 
1.3 
1.4} 
1,2 
1.4 
1.4 
1.3 
1.5 
1.4 
1.4 
1.5 
1.2 



.51 
.48 
.50 
.48 
.50 
.41 
.49 
.49 
47 
.50 
.49 
.49 
.49 
.49 



5.0 
4.5 
4.6 
5.0 
4.3 
3.8 
4.6 
4.1 
4.3 
4.5 
4.2 
4.2 
4.5 
4.8 



.00 
.83 
.80 
.00 
.97 
.99 
.74 
.97 
.93 
.37 
.96 
.96 
.85 
.60 



composite scores 
nor differences 1 



11.8 
11.7 
12.0 
11.6 
11.8 
12.0 
12.2 
11.0 
11.1 
11.4 
11.0 
11.9 
11.9 
12.6 



1.00 
.78 
.74 
.93 
.84 
.79 
.70 
1.00 
1.00 
1.10 
1.10 
.78 
.95 
.24 



.0 
.2 
.3 
.2 
.2 
.3 
.0 
.5 
.3 
.4 
.3 
.4 
.2 
8 



.00 
.44 
.49 
.45 
.45 
.46 
.10 
.70 
.48 
.50 
.48 
.49 
.44 
.40 



22.2 
20.5 
21.2 
20.4 
21.2 
20.8 
20.7 
20.0 
20.4 
20.7 
20.0 
20.8 
21.3 
22.4 



3.3 
2.0 
2.6 
2.0 
2.4 
1.6 
2.5 
2.3 
2.4 

;..8 

2.3 
2.1 
2.3 
3.0 



(Mechanical, Administrative, General, and Electronics) were obtained from 
n the selection AI and composite scores. 
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APPENDIX 6: TRAINING OUTCOME VARIABLE MEANS AND STANDARD DEVIATIONS BY COURSE 



Quality of 



Academic 



Nonacad. 



SIA tiie 



Retraining 
time 



Academic 
attrition 



Nonacad. 
attrition 



Chau:te Technical Training Center 
Aircraft Envlromental Systems Mechanic 
Aircraft Fuel Systems Mechanic 
Airframe Repair Specialist 
Missile Maintenance Specialist 
Special Vehicle Mechanic 
F1re Protection Specialist 
Aircraft Electrical Sys. Specialist 
Jet Engine Mechanic 

Sheppard Technical Training Center 
Aircraft Loadmaster 
Telephone Switching Specialist 
Cable Splicing Specialist 
Tactical Aircraft Maint. Specialist 
Electrician 
Carpentry Specialist 
Financial Service Specialist 
Medical Services Specialist 
Surgical Services Specialist 
Medical Administrative Specialist 
Physical Therapy Specialist 
Dental Assistant Specialist 
Financial Management Assistant 
Medical Laboratory Specialist 

Lackland Military Training Center 
Security Specialist 
Law Enforcement Specialist 



AFSC 




X 


SD 


X 


SO 


X 


_SD X_ 


SD 


X 


SD 


X 


SD 


X 


_s 


42331 


M a 


86.6 


5.5 


61.9 


30.1 


6.80 


8.0 .40 


.88 


29.2 


62.5 


.03 


.17 


.00 


.00 


42333 


M 


07.5 


6.4 


1.5 


2.6 


.62 


1.3 .20 


.62 


11.6 


26.5 


.00 


.00 


.00 


.00 


42735 


M 


84.9 


8.1 


21.3 


13.4 


.46 


1.2 .09 


.29 


9.1 


25.0 


.01 


.11 


.00 


.00 


44330 


M 


89.1 


4.8 


5.8 


5.2 


2.10 


2.3 .09 


.34 


.0 


.0 


.00 


.00 


.00 


.00 


47231 


M 


88.0 


6.0 


52.8 


25.7 


.83 


3.1 .00 


.00 


6.5 


37.0 


.03 


.17 


.00 


.00 


57130 


M 


89.8 


5.7 


2.5 


3.5 


.68 


1.6 .10 


1.00 


18.9 


44.1 


.04 


.19 


.00 


.00 


42330 


V 


90.0 


6.3 


3.8 


5.5 


2.30 


3.1 .10 


.58 


8.8 


38.0 


.02 


.14 


.00 


.00 


42632 


V 


85.0 


8.9 


4.6 


6.5 


.70 


2.7 .16 


1.70 


16.5 


60.2 


.03 


.17 


.00 


.00 


11430 


M 


87.5 


6.1 


4.2 


4.9 


.57 


1.10 .02 


.16 


14.1 


37.5 


.03 


.16 


.00 


.00 


36231 


M 


83.5 


5.1 


8.2 


8.1 


1.90 


4.20 .09 


.35 


45.5 


151.0 


.07 


.26 


.00 


.00 


36131 


M 


81.9 


13.0 


6.4 


6.1 


.77 


.80 .05 


.22 


16.2 


65.6 


.03 


.16 


.00 


.00 


43131 


M 


82.7 


7.0 


.0 


.0 


.00 


.00 .00 


.00 


.0 


.0 


.00 


.00 


.00 


.00 


54230 


M 


87.9 


6.7 


1.5 


2.8 


.11 


.47 .02 


.1. 


2.9 


13.6 


.00 


.00 


.00 


.00 


55230 


M 


84.4 


4.9 


1.2 


4.5 


.09 


.30 .00 


.00 


3.3 


11.6 


.00 


.00 


.00 


.00 


67.132 


M 


84.8 


6.2 


3.9 


5.1 


.20 


.65 .00 


.00 


2.4 


13.3 


.00 


.00 


.00 


.00 


90230 


M 


82.2 


5.7 


16.7 


26.2 


1.60 


3.00 .15 


.53 


4.2 


24.8 


.00 


.00 


.01 


.06 


90320 


M 


85.3 


6.6 


8.1 


16.5 


1.40 


2.20 .32 


.83 


1.0 


7.0 


,00 


.00 


.00 


.00 


90630 


M 


83.2 


5.5 


7.5 


19.9 


1.60 


5.90 .38 


1.80 


7.1 


37.1 


.03 


.16 


.01 


.08 


91330 


M 


77.8 


17.1 


2.2 


3.5 


1.60 


2.30 .12 


.42 


3.9 


26.9 


.08 


.28 


.05 


.22 


98130 


M 


87.9 


5.1 


6.4 


12.6 


2.40 


2.50 .04 


.20 


13.7 


62.8 


.02 


.18 


.01 


.11 


67231 


V 


86.4 


6.7 


10.3 


9.1 


1.30 


4.20 .05 


.29 


25.0 


70.5 


.06 


.23 


.00 


.00 


92430 


V 


84.9 


■ 5.1 


24.4 


24.6 


2.50 


3.50 .11 


.39 


53.7 


131.5 


.11 


.32 


.00 


.00 


81130 


M 


79.4 


6.7 


.54 


1.5 


1.5 


3.3 .51 


3.2 


8.20 


40.8 


.03 


.16 


.01 


.08 


81132 


M 


80.1 


6.1 


1.20 


3.4 


1.3 


3.2 .50 


2.9 


.61 


5.5 


.02 


.12 


.02 


.15 
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APPENDIX G: (Concluded) 



Quality of 

performance SIA time 



Academic Nonacad. 
counseling counseling 



Retraining Academic Nonacad. 
time attrition attrition 



Lowry Technical Training Center AFSC 

Continuous Photoprocessing Specialist 23330 M a 

Instrumentation Mechanic 31633 M 

Avionic Sensor Systems Specialist 32232 M 

Precision Measuring Equip. Specialist 32430 M 

Aerospace Ground Equipment Specialist 32630 M 

Computerized Test Station Specialist 32634 M 

Attack Control Systems Specialist 32636 M 

Munition Systems Specialist 46130 M 

Material Facilities Specialist 64531 M 

Armament Systems Specialist 46230 V 

Keesler Technical Training Center 

Command and Control Specialist 27430 M 

Wideband Comm. Equip. Specialist 30430 M 

Electronic Computer Specialist 30534 M 

Telecommunications Control Specialist 30730 M 

Airborne Warning Radar Specialist 32832 M 

Electronic Warfare Radar Specialist 32833 M 

Computer Specialist 61130 M 

Administration Specialist 70230 M 

Personnel Specialist 73230 M 

Personal Affairs Specialist 73231 M 

Ground Radio Operator 29333 V 

Aircraft Warning Radar Specialist 30332 V 

Navigation Systems Specialist 32831 V 

Computer Programming Specialist 51131 v 



a M « Course included in model -development sample; 



89.7 
89.4 
91.2 
90.3 
89.4 
91.2 
86.9 
89.5 
87.2 
87.7 
86.6 
89.5. 
90.5 
87.9 
V - 



X 


SD 


"x 


SD 






y 

A 




V 
A 


SD 


X 


SD 


_X_ 


s 


82.2 


8.5 


2.4 


~Ta 


7756 


3.90 


.02 


.14 


15.40 


63.50 


.06 


.23 


.00 


.0 


89.9 


4.8 


22.6 


25.1 


1.80 


4.40 


.00 


.00 


19.60 


79.40 


.12 


.33 


.00 


.0 


84.9 


7.8 


3.4 


5.3 


.78 


1.20 


.03 


.17 


6.80 


21.40 


.00 


.00 


.00 


.0 


84.2 


5.0 


18.0 


16.5 


.99 


1.20 


.01 


.08 


.11 


1.20 


.01 


.12 


.00 


.0 


90.3 


4.5 


4.2 


5.9 


.62 


1.60 


.00 


.00 


22.60 


116.10 


.05 


.21 


.00 


.0 


92.6 


3.9 


8.6 


15.1 


.40 


.70 


.15 


.36 


.00 


.00 


.00 


.00 


.00 


.0 


84.5 


5.1 


9.6 


12.8 


1.30 


2.40 


.12 


.79 


20.61 


61.58 


.06 


.24 


.00 


.0 


84.5 


8.7 


2.6 


4.0 


1.20 


4; 50 


.05 


.47 


11.30 


59.00 


.04 


.20 


.00 


.0 


86.4 


6.2 


1.0 


1.6 


.16 


.50 


.01 


.07 


2.00 


11.00 


.00 


.00 


.00 


.0 


85.9 


7.5 


1.5 


2.6 


.89 


2.70 


.07 


.40 


14.90 


78.90 


.05 


.20 


.00 


.0 



5.7 
4.5 
4.5 
4.6 
4.1 
4.2 
7.8 
6.0 
6.4 
5.6 
5.6 
4.4 
4.4 
6.9 



2.30 8.4 
6.30 6.6 
3.60 7.0 
2.60 o.O 
5.50 10.1 
6.20 16.5 



.29 
1.30 
1.60 
.38 
12.50 
10.90 
3.30 



1.0 
3.4 
6.0 
1.8 
14.4 
9.2 
8.8 



5.80 19.8 



.19 
3.00 
2.80 
1.80 
3.20 
3.10 

.35 
1.00 
1.10 

.16 
2.80 
2.10 
1.40 

.60 



.67 
3.60 
3.50 
2.60 
7.50 
4.00 
1.40 
4.60 
1.60 

.44 
4.40 
3.20 
1.80 
1.30 



.00 
.09 
.03 
.07 
.00 
.16 
.01 
.04 
.05 
.00 
.03 
.06 
.05 
.02 



.00 
.33 
.17 
.38 
.00 
.47 
.07 
.28 
.28 
.00 
.20 
.23 
.22 
.14 



.00 
27.30 
15.80 
9.00 
15.70 
18.20 
2.00 
7.50 
5.60 
1.70 
28.00 
13.90 
14.00 
9.20 



Course included 1n cross-validation sample. 



.00 
60.60 
37.50 
23.30 
53.80 
70.90 
10.40 
30.30 
20.00 
14.50 
66.20 
41.00 
31.70 
22.30 



.00 .00 

.03 .16 

.01 .10 

.02 .15 

.10 .30 

.02 .13 

.01 .07 

.02 .15 

.02 .12 

.00 .00 

.06 .24 

.04 .18 

.01 .11 

.00 .09 



.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 

.00 .0 
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APPENDIX H ; INTERCORRELATIONS OF STUDENT INPUT, COURSE CONTENT, 
AND TRAINING OUTCOME VARIABLES IN THE TOTAL SAMPLE 



Variables* Variables 



SI-1 SI-2 SI -3 SI-4 SI-5 SI-6 SI-7 SI-8 SI-9 SI-10 SI-11 SI-12 CC-1 CC-2 CC-3 CC-4 Cfrj CC^6 

SI-1 Aptitude Selector 1.00 .48 . 48 . 73 . 64 . 51 .35 -.17 -.12 .18 . 09 .18 .27 -.23 -.13 . 01 .14 -.12 

- SI-2 Mechanical 1.00 .06 .47 . 66 . 42 . 26 -.04 . 00 . 08 . 02 .10 .30 -.07 -.25 -.14 .18 . 04 

Sl-3 Administration 1.00 .45 .29 .35 .27 -.05 -.05 .17 .08 .11 .06 -.14 .00 .05 .02 -.13 

SI-4 General 1-00 - 76 « 66 -36 -.09 -.04 .20 .15 .20 .25 -.13 -.07 -.02 .06 -.13 

SI-5 Electronics 1-00 .52 .41 -.07 -.01 .19 .10 .18 .36 -.15 -.18 -.08 .15 -.04 

SI-6 Reading Level 1-00 .26 -.05 .00 .18 .12 .19 .19 -.09 -.06 -.02 .05 -.09 

SI-7 Academic Motivation 1.00 -.06 -.04 .31 .25 .26 .21 -.12 -.08 -.02 .04 -.01 

SI-8 Simple Interest 1-00 .70 -.04 -.04 -.09 -.01 .14 .04 -.08 -.02 .07 

SI-9 Preference Interest 1.00 -05 .00 -.03 .00 .11 -.01 -.08 .01 .04 

SI-10 Educational Level 1-00 .13 .44 .08 -.05 -.02 -.02 .05 -02 



SI-11 Educational Preparation 



.51 .09 
1.00 .40 -.19 -14 .04 



1.00 -.03 -.31 
1.00 .03 
1.00 



1.00 .10 .00 .27 .37 .19 -.10 .16 
. 1.00 .15 -.11 -07 -.01 .09 -.05 

.S? Course Length !•» -.36 -.40 -.17 

CC-2 Day Length 

CC-3 Student-Faculty Ratio '- 00 

CC-4 Instructor Quality 

CC-5 Instructor Experience 

; CC-6 Aids In Use 

CC-7 Hands-On Practice 

> CC-8 Amount of Feedback 

CC-9 Practice 

; CC-10 Reenllstment Bonus 

CC-11 Student Flow 

CC-12 Ocupatlonal Difficulty 

CC-13 Reading Difficulty 

[. CC-14 Diversity 

; CCrl5 Abstract Knowledge 

! CC-16 Expected Attrition 

; TO-l Quality of Performance 

\ TO-2 SIA Time 

TO-3 Academic Counseling 

! TO-4 Nonacadmelc Counseling 

j TO-5 Retraining Time 

; TO-6 Academic Attrition 

: TO-7 Nonacademic Attrition 

^ ERJC 
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APPENDIX H : (Concluded) 



tables* ! Variables 

i ci * CC-7 CC-8 CC-iT cC-10 CC£1_1 CM2 CC^U CC£14 CMS CC^lj TO-1 TO-2 TO-3 TO-4 TO-5 TO-6 TO-7 

SM Aptitude Selector -.24 .04 -10 .00 -.24 .04 .25 .28 ~ ST TsT ^03 ^15 ^06 "^09 ^08 Tfi 

" • ' S 'I?' -.11 -.05 -.08 .23 -.10 .34 .06 .31 .28 .18 .26 .00 -. 11 - 02 - - 8 -io 

SI-3 Administration -.12 .02 -.03 -.09 -.13 -.13 .17 .07 .03 .09 .25 -.05 -.13 -.06 -.07 -.06 -03 

S -4 General -.2 -.01 -.11 .07 -.17 .04 .13 .28 .26 .17 .34 -.'04 -.14 -.04 -.04 -.08 —02 

SU^Tf* i -21 -02 -.13 .17 -.17 .21 .15 .38 .34 .24 .34 -.01 -.11 -.05 -.07 - 08 - 04 

SI-6 Readlnlg Level -.18 -.04 -.08 .05 -.13 .06 .10 .20 .20 .15 33 - 05 - lfi 02 m no ni 

«-7 Academic Motivation -.12 -.01 -.08 .04 -.12 .06 .13 £ 5 Al S - 5 - - 3 - -£ 

S -8 Simple Inte , est . 05 , #15 >12 #15 #25 02 _ >03 _ >06 _ >o8 oQ _ >og _ >()4 u 04 

SJ-9 Preference Interest -.01 -.12 .07 .11 .14 .06 -.01 -.02 .00 -.03 -.05 -.06 -.01 .01 -.01 01 00 

S-J Educational Level -.09 -.02 -.04 -.02 -.10 -.01 .07 .09 .08 .10 .15 00 - 03 - 03 00 -M "2 

«- Educational Preparation .04 .06 -.28 -.16 -.18 -.11 -.04 .09 .43 .12 01 7 - \o2 0 * 

2" J 9 ' » *h -10 .00 -.08 .00 -.16 .03 .13 .15 .15 .14 .16 .02 -.01 -.02 .02 .0 -.'o 

,CC-1 Course Length -.30 -.10 -.08 .24 -.29 .50 .37 .89 .60 .66 .20 .15 . 11 -.03 .07 01 -02 

*C-2 Day Length .07 -08 .24 .15 .38 -.03 -.54 -.36 -.14 -.27 -.26 - 01 - 0 06 - 0 01 04 

CM Student-Faculty Ratio .07 .04 .11 -.23 .25 -.55 -30 -.36 -.17 -28 -20 0 , ! ! - £ £ 

CC-4 Instructor Quality .04 .54 -37 -50 -.21 -.31 .19 -.09 .03 -10 .08 -.01 -05 - 02 - 06 - 01 00 

1 ?f it Expor,ence -f • 00 - 18 - 06 - 40 - 59 - 25 - 52 - 3o - 24 -z« o - :2 : 

CM Jlds in Use .45 -04 -11 .31 .28 .16 .17 .16 .13 .06 .00 .15 .05 .00 .01 05 - 04 

CC-7 Hands-On Practice 1.00 -07 .18 .05 .45 -18 .03 -.33 -.14 .02 -.09 .09 .01 .02 0 01 - 01 

SIod J J .» 1,00 * 14 * 49 - 07 - 31 - 40 -' 35 - 07 -« 24 "'I 2 -04 .06 -03 .00 .05 

CC- 1 Student Flow lt0 0 -.20 -.23 -.42 -.46 -.23 -.30 -.20 .00 .08 - 04 - 01 02 

CC-12 Ocupatlonal Difficulty i 00 03 53 4fi m no ™ I «! «, 

CC-13 Reading Difficulty 100 /£ 'l! 'J? \° •?? '° 4 •« '° '01 '00 

CC-14 Diversity 1,00 .'2 '?! * 43 ^ -01 -06 .01 .01-04 



CC-15 Abstract Knowledge 



.25 .17 .10 -.04 .09 .03 -.03 



TO-1 Quality of Performance h0 ° ,*° 9 -?° 'JJ "'J 2 ^ -01 

T0-2 SIA Time 1,00 " # JJ "- 3 ° -- 09 -' 20 - 22 -07 

TO-3 Academic Counseling 100 /" •« - 12 -« 

TO-4 Nonacadmelc Counseling , !? tf ; 52 - 05 

TO-5 Retraining Time 1,00 ,* ,16 * 43 

TO-6 Academic Attrition 1,0 ° * 68 

TO-7 Wonacademlc Attrition ,0 ° 

'Student input variable are denoted by the abbreviation SI. Course Content variables are denoted by the abbreviation" 
CC. Training outcome variables are denoted by the abbreviation TO. 

Note. Correlations are based on 5,981 students and 48 courses. 
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APPENDIX I ; INTERCORRELATIONS OF STUDENT INPUT, COURSE CONTENT, AND 

TRAINING OUTCOME VARIABLES IN THE CROSS-VALIDATION SAMPLE 



Variables* 

SI-1 Aptitude Selector 

SI-2 Mechanical 

SI-3 Administration 

SI-4 General 

SI-5 Electronics 

SI-6 Readlnlg Level 

SI-7 Academic Motivation 

SI-8 Simple Interest 

SI-9 Preference Interest 

SI-10 Educational Level 

SI-1 1 Educational Preparation 

SI -12 Age 

CC-1 Course Length 

oo CC-2 Day Length 



u> 



Variables 


SI-1 SI-2 SI-3 SI-4 SI-5 SI-6 SI-7 SI-8 SI-9 SI-' 


10 SI-11 


SI-12 CC-1 


CC-2 CC-3 CC-4 CC-5 CC-6 


i nn cn ac ci 

I.UU .t)U mHD .61 


CO 
• DO 


• 39 


.28 


.00 


.01 


.16 


.07 


.17 


.25 


-.17 


.00 


-.15 


.07 


-.12 


1 nn ao 

I.UU . 10 » He 


CC 

• DO 


• 39 


• 25 


ft A 

.04 


.13 


.02 


-.03 


.08 


.24 


-.12 


-.36 


.18 


.29 


-.06 


I.UU .bo 


.34 


.35 


0*7 

.27 


.10 


.00 


.23 


.14 


.17 


.01 


-.06 


.18 


-.23 


-.15 


-.08 


uuu 


• to 


cc 
.65 


Oft 

• 39 


ft n 

.08 


.03 


.25 


.21 


.26 


.29 


-.15 


-.06 


-.09 


.03 


.03 




1.00 


.52 


.43 


.06 


.06 


.22 


.14 


.21 


.38 


-.17 


-.18 


.00 


.15 


.00 






1.00 


.28 


.10 


.09 


.20 


.17 


.16 


.19 


-.11 


-.09 


.02 


.04 


.06 








1.00 


.03 


.03 


.35 


.33 


.37 


.18 


-.02 


.04 


-.08 


-.06 


.11 










1.00 


.63 


.05 


-.02 


-.05 


.00 


-.06 


-.02 


.04 


.02 


.05 












1.00 


.02 


.01 


.00 


-.09 


.GO 


-.14 


.13 


.02 


.00 














1.00 


.27 


.49 


.05 


.01 


.08 


-.12 


-.10 


.13 
















1.00 


.22 


.05 


.41 


.28 


-.2.1 


-.63 


.71 


















1.00 


.14 


-.05 


.00 


-.09 


.01 


.04 




















1.00 


-.55 


-.24 


.08 


.48 


.16 






















1.00 


.41 


-.43 


-.71 


.32 



CC-3 Student-Faculty Ratio 1>0 0 -.73 -.69 

CC-4 Instructor Quality " i.qo * 



.34 

^ , T . — * 27 --16 

CC-5 Instructor Experience 1 00 - 44 

CC-6 Aids In Use # ^ 00 

CC-7 Hands-On Practice 

CC-8 Amount of Feedback 

CC-9 Practice 

CC-10 Reenllstment Bonus 

CC-11 Student Flow 

CC-12 Occupational Difficulty 

CC-13 Reading Difficulty 

CC-14 Diversity 

CC-15 Abstract Knowledge 

CC-16 Expected Attrition 

T0-1 Quality of Performance 

TO-2 SIA Time 

TO-3 Academic Counseling 

TO-4 Nonacadmelc Counseling 

TO-5 Retraining Time 

TO-6 Academic Attrition - 

TO-7 Nonacademlc Attrition 107 
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APPENDIX I : (Concluded) 
Variables* Variables 



CC-7 CC-8 CC-9 CC-10 CC-11 CC-12 CC-13 CC-14 CC-15 CC-16 TO-1 TO-2 TO-3 TO-4 TO-5 TO-6 TO-7 



SI -1 


Aptitude Selector 


-.15 


.24 


.06 


.02 


-.03 


-.27 


13 


93 


04 

. UH 


3ft 


33 
. 00 


- Oft 

— . uo 


1 A 
-• l*f 


nn 
.UU 


1C 

-• Id 


-• 15 


nn 
• 00 


SI-2 


Mechanical 


-•22 


.37 


-.26 


.41 


.00 


.31 


.10 


.27 


.09 


- 15 


39 

. <JC 


— . £0 


m 94 


00 
. UU 


- 90 

— • C\J 


10 

-• iy 


00 
• UZ 


SI -3 


Administrative 


•02 


-.01 


.19 


-.23 


-.03 


-.43 


.06 


.01 

• U 1 


-.03 


44 
. tt 


94 

. CH 


- 01 
• U 1 


- 10 
-. IU 


- fifi 
-. UO 


n7 

-•U/ 


oo 
-•Uo 


n7 
-•U/ 


SI -4 


General 


-.16 


.30 


-.11 


.12 


.00 


-. 14 


91 


34 


99 


49 

• HC 


3Q 
. oy 


- Oft 

— . uo 


_ on 
-. tU 


0.0 
-.UZ 


1 Q 

-• Io 


-• lb 


no 
-•UZ 


SI -5 


Electronics 


-.23 


.41 


-.17 


.28 


.04 


-.01 


. 18 


41 


94 


30 


40 

• HU 


• 15 
-. 10 


-.01 

-•Z 1 


oo 
-•Uo 


IO 

-• Iy 


IO 

-• Io 


no 
-.03 


SI-6 


Readlnlg Level 


-.14 


.18 


-.14 


.10 


.00 


.00 


.17 


.24 


lfi 
• i u 


91 


37 

.0/ 


- 19 
— . \C 


„. pc 
— . tO 


01 
-•U 1 


_ oo 
-. CO 


Ol 

-•Z 1 


nn 
.00 


SI -7 


Academic Motivation 


-.10 


* .10 


-.04 


.06 


.08 

. UW 


-.OQ 


08 


90 


18 


97 
• CI 


91 
• C 1 


-. U 1 


-•U/ 


oo 
.UU 


m 
-•Ul 


oo 
-.02 


m 

-.01 


SI -8 


Simple Interest- 


-.10 


-.05 


-.06 


-.03 


.03 


.01 


.12 


.04 


-.01 


.00 


.05 


- 06 


- 05 
— . uo 


03 
. uo 


- 09 
-. Ut 


00 
.UU 


no 
-•Uo 


SI -9 


Preference Interest 


-.02 


-.01 


-.14 


.03 


.01 


.13 


.08 


-.03 


-.03 


- 10 


OQ 
. uy 


- OQ 
• uy 


• 19 

-. \c 


01 
. U 1 


n7 
-•U/ 


OA 
-•UH 


m 
• Ul 


SI-10 


Educational Level 


-.03 


.00 


-.04 


-.03 


-.01 


-. 10 

. IU 


.OQ 


10 


10 


99 
. cc 


1 1 
• I I 


09 

• KJC 


09 
-. \JC 


oo 
-•UZ 


nn 
• UU 


-•01 


-.01 


SI-11 


Educational Preparation 


.10 


-.15 


-.30 


.20 


.71 


01 

• U 1 


04 

. U"? 


. 18 

. IU 


fifi 


47 

• H/ 


• 03 
— • UO 


31 
.0 1 


HA 
. UH 


oo 

• UZ 


no 
• Uy 


no 
• Oo 


• 01 


SI-12 


Age 


-.03 


.16 


-.06 


.07 


-•01 


-•09 


10 


1 fi 


14 

. IH 


9fi 

. CO 


15 
• 10 


_ no 
-. Ut 


C\0 
• \JC 


Ofi 
• UD 


oo 
• Uo 


nn 
• 00 


• 07 


CC-1 


Course Length 


-.57 


.72 


-.25 


.42 


-.09 


.08 


.41 


Q4 


54> 


44 

. HH 


1Q 
• iy 


05 
. UO 


OK 
. UO 


01 
-•U 1 


oo 
.UU 


m 
-.01 


no 
-.02 


CC-2 


Day Length 


• 41 


-•46 


.13 


•22 


.54 


14 


- 7fi 
— • / u 


fifi 
. DO 


on 


• 3fi 

-.00 


- 1A 
-• IH 


IE 

. Io 


oc 
.Uo 


oo 
• UZ 


• 06 


n4 

• 04 


• 02 


CC-3 


Student-Faculty Ratio 


.57 


-.66 


.68 


-.64 


• 09 


-.72 


-.44 


- 36 

• «JU 


. <JV 


4ft 
.HO 


- 07 
-• U/ 


31 
.0 1 


10 


Ol 

-•Ul 


1 o 
• IZ 


in 

• 10 


no 

-.02 


CC-4 


Instructor Quality 


-.27 


. 18 


-.50 


.06 


08 


OQ 

. Uy 


45 


1Q 


. 1 1 


-• iy 


• Uo 


-•Zo 


OO 

-•Uy 


n i 
• 01 


no 

-.08 


-.04 


• 00 


CC-5 


Instructor Experience 


-.53 


.67 


-.19 


.44 


-.64 

. U*t 


Ofi 
. UU 


37 
• o / 


44 

*HH 


• 90 


- 4Q 

— .ny 


1/1 


01 

-•o 1 


1 0 

-• Io 


no 


-. 12 


-.18 


• 01 


CC-6 


Aids In Use 


-.03 


-.28 


-.43 


.46 




57 


Ifi 


34 
. OH 


74 

• /H 


. iy 


-• 1 1 


01 
• 0 1 


• U/ 


• 02 


.14 


• 11 


• 00 


CC-7 


Hands-On Practice 


1.00 


-.42 


.52 


-.48 


- 01 


— .HI 


- 5? 


« 79 


. nT 

-•u 1 


in 


07 
-•U/ 


oo 
• UZ 


O/l 

• U4 


no 

-.02 


no 

• 02 


• 01 


-.01 


CC-8 


Amount of Feedback 




1.00 


-.34 


.66 


-.98 


Ol 

• Cl 


9fi 


fifi 
. DO 


99 


OR 


OO 
• CV 


1 c 
-• ID 


-•U/ 


n i 

-.01 


no 

-•09 


no 

-.09 


• 00 


CC-9 


Practice 






1.00 


- 58 
. «?o 


-. 1? 


- 78 
• /o 


- 5ft 

— . UO 


. 59 


-.HO 


AO 

•ny 


OO 

• Uo 


• UH 


1 0 


no 

-.03 


nn 

• 00 


• 00 


-.01 


CC-10 


Reenllstment Bonus 








1.00 


.27 


.70 


10 

• 1 u 


5? 


37 
.0/ 


• 53 


Oft 
• Uo 


1C 

-• 10 


-•Uo 


oo 
• Oo 


no 

-•08 


no 
-.08 


no 

• 02 


CC-11 


Student Flow 










1.00 
i . uu 


03 


- 93 


• Ofi 
— . UO 


30 


in 


OO 

• Uo 


oc 
• UO 


• Uo 


• 01 


no 

• 02 


• 02 


• 00 


CC-12 


Ocupatlonal Difficulty 












1 00 


07 


95 


15 


- 09 

-• yc 


07 
-•U/ 


oo 
-•cc 


-• 15 


nc 

• 05 


-•06 


-.04 


.04 


CC-13 


Reading Difficulty 














1.00 


.77 


.95 


69 


04 

• UH 


- 09 

— • \JC 


• 07 

-• u/ 


oo 

• UU 


on 
• UU 


on 
• UU 


m 

-.01 


CC-14 


Diversity 
















1 00 


fift 

. DO 


fil 

. 0 1 


IK 
• 10 


07 
• U/ 


m 
• Ul 


on 
• 00 


n t 

.01 


• 00 


-.02 


PP- 15 

WW IJ 


MDstrdCL Nnuwicagc 


















1.00 


• 55 


• 05 


.21 


• 08 


• 00 


.07 


•05 


-.03 


CC-16 


Expected Attrition 




















1.00 


.09 


• 29 


.16 


-.03 


.08 


.07 


-.06 


TO-1 


Quality of Performanre 






















1.00 


-.38 


-.54 


-.09 


-.46 


-.40 


-.07 


TO-2 


SIA Time 
























1.00 


.51 


.07 


.37 


.33 


.01 


TO-3 


Academic Counseling 


























1.00 


• 16 


.69 


.69 


.12 


TO-4 


Nonacademlc Counseling 




























1.00 


.07 


.02 


.93 


TO-5 


Retraining Time 






























1.00 


.78 


.02 


TO-6 


Academic Attrition 
































1.00 


.00 


TO-7 


Nonacademlc Attrition 
































i nn 



— ^ . . ;.i 

^Student Input are denoted by the abbreviation SI. Course Content variables are denoted by the abbreviation CC. Training 
outcome variables are denoted by the abbreviation TO. 



Note . Correlations are based on 890 students and 9 courses. 



APPENDIX J: REFINED MODEL WITH REGRESSION COEFFICIENTS 




COURSE CONT ENT VARIABLES 
.48 



DIVERSITY 



expected attrition 



CS 



ABSTRACT KNOWLEDGE 




READING DIFFICULTY 



SUBJECT MATTER DIFFICULTY 



inl 



OCCUPaTKDNALDFFICULTY 



6 4 1 .38 1 .02^ . 3^ -.05^. 39^ -.27 ^.08^ -.35^ -.65^42^ .16^-.19 .06^ .03^ 



05T .04T -.03 ▼ -.04V -01 ▼ -.02 




TRAINING 
.16.05 -^04 OUTCOME 
VARIABLES 

.23 



J ACADEMIC COUNSEL NG 



p J ACADEMIC ATT 

31^37 , ^ 

- 1 RETRAfJfKa 



ATTRITION 



NONACADEMC 
COUNSEUNG 



NONACADEMC ATTRITION 



no 



in 



APPENDIX K: REFINED MODEL WITH TOTAL EFFECTS 



COURSE CO NTENT VARIABLES 

48 




DIVERSITY 



EXPECTED ATTRITION 



ABSTRACT KNOWLEDGE 



T53. 



READWG DTFICULTY 



SUBJECT MATTER DIFFICULTY 



I 



52. 



^7T 



OCCUPATIONAL DFFCULTY 



I -.14 fi3 




OUTCOME 



OUAUTYOF ft 
PERFORMANCE 



SIA TIME 

^ .23 



TRAINING 
OUTCOME 
VARIABLES 



ACADEMIC COUNSELING 



ACADEMIC ATTRITION | 

ill 



RETRAINING 



NONACADEMC 
COUNSELING 






NONACADEMC ATTRfriON | 





' o 12 
ERIC* 
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APPENDIX L : CORRELATED ERROR TERMS AM'NG COURSE VARIABLES 



Student- 



Instruc- 



Course content 


Course 


Day 


faculty 


Instructor 


Instructor 


tional 


Feedback 


variables* 


length 


length 


ratio 


quality 


experience 


support 


Intensity 


' Day Length 


.02 


.63 












Student-Faculty Ratio 


.02 


.23 


.62 










"Instructor Quality 


.12 


.00 


.12 


.52 








Instructor Experience 


.02 


.13 


-.24 


.01 


.65 






instructional Support 


.14 


.01 


.16 


.00 


.16 


.63 




Feedback Intensity 


.54 


.38 


.04 


.00 


.05 


.03 


.64 



a All observed scores are assumed to be a function of true plus error variance. Correlations represent projected 
relationships among errors of observations on the variables. 
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APPENDIX M : CORRELATED ERROR TERMS OF COURSE CONTENT AND TRAINING OUTCOME VARIABLES 



Outcome Variables 

Assessed SIA Academic Nonacademlc Retraining Academic Nonacademlc 

quality time counsel 1 ng counsel Ing time attrition attrition 

-•08 -.08 .14 .03 JO -.02 .00 



Jbay Length 


-.12 


.01 


-.03 


.11 


-.13 


.01 


.14 


■ Student-Faculty Ratio 


-.05 


-.02 


-.02 


-.03 


-.12 


.01 


.60 


r . 

^Instructor Quality 


.02 


.00 


.01 


.01 


.01 


.00 


.00 


' Instructional Support 


-.03 


.05 


.05 


-.01 


.01 


.00 


-.08 


Feedback Intensity 


-.07 


.02 


.00 


-.03 


.29 


-.04 


.00 



i a All observed scores are assumed to be a function of true plus error variance. Correlations represent projected 
' relationships among errors of observations on the variables. 



115 



Course content 
L variables* 
Course Length 



APPENDIX N: CORRELATED ERROR TERMS AMONG TRAINING OUTCOME VARIABLES 



r ^Training 
} outcone 
t: variables 8 


Assessed 
quality 
performance 


SIA 
time 


Academic 
ccunsellng 


Nonacademlc 
counseling 


Retraining 
time 


Academic Nonacademlc 
attrition attrition- 


Assessed Quality of 
7 JP erf ormance 


.46 












SI A Time 


-.16 


.87 










Academic Counseling 


-.37 


.32 


.63 








•Nonacademic Counseling 


-.10 


.on 


.23 


.75 






Retraining Time 


-.09 


.12 


.55 


.17 


.47 




Academic Attrition 


-.14 


.04 


.34 


-.09 


.34 


.30 


Nonacademlc Attrition 


-.12 


.01 


.11 


.59 


.04 


-.03 .64 



a All observed scores are assumed to be a function of true plus error variance. Correlations represent projected 
Relationships among errors of observations on the variables. 
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